[PATCH cluster/ha-manager/perl-rs/proxmox v2 00/40] dynamic scheduler + load rebalancer

[Daniel Kral <d.kral@proxmox.com>]

This RFC series proposes an implementation for a dynamic scheduler and
manual/automatic static/dynamic load rebalancer by implementing the
following:

- gather dynamic node and service usage information and use it in the
  dynamic scheduler, and

- implement a load rebalancer, which actively moves HA resources to
  other nodes, to lower the overall cluster node imbalance, while
  adhering to the HA rules.



== Changelog v1 -> v2 ==

The following changelog is mostly an overview of the more important
changes and fixes. I've added per-patch changelogs, but in general the
patch messages and documentation was improved for most of them.

I haven't fully updated the cover letter description, but regarding the
benchmarks I quickly ran the same benchmarks I wrote for the initial RFC
series and the numbers didn't change quite that much. I intend to
contribute these benchmarks to the main repository so we can have a
better eye on that, but they need some more care to be truly a benefit
to the repositories.

All patches were individually built with the packages derivations of
`git rebase master --exec 'make clean && make deb'` and special care was
taken for the deprecation of proxmox-resource-scheduling items used in
the pve-rs bindings.

Otherwise it would be great to have at least a few more fresh eyes
testing the functionality from this series. I'll do the same over the
next few days.

I've dropped the pve-manager patches as having the core functionality
out is more important now, but the pve-manager from v1 should be fully
functional for this. I'll send a follow-up to this (or include it in a
v3) accompanying pve-docs patches as well.

Thank you very much for the feedback, @Dominik, @Thomas, and
@Maximiliano!

(important) fixes v1 -> v2:
- make the dynamic load balancer assume starting resources are using
  their `max*` stats, so that rebalance on start works as expected and
  doesn't choose the least loaded node on startup
- do not panic of `topsis::score_alternatives(...)` is called with no
  alternatives
- implement the `Ord` trait for `ScoredMigration` correctly and also
  sort by `Migration` if two imbalance scores are (nearly) equal
- fix floating-point approximation errors for imbalance scores for
  `ScoredMigration`
- fix bug where resource bundles could have been created, where
  dependent resources were still in migrate or relocate state (but
  wouldn't really happen as for now it is not called at all if there is
  any migration in the HA Manager state)

(important) changes v1 -> v2:
- rebase all repos on the current master
- drop pve-manager patches for now, will follow up on this (+ with
  pve-docs patches regarding this feature)
- move the non-HA resource accounting out into its own series [0]
  (which is already applied!)
- rename `proxmox_resource_scheduling::scheduler::ClusterUsage` to
  `proxmox_resource_scheduling::scheduler::Scheduler`
- move the `Usage` implementation to the proxmox-resource-scheduling:
  reduce code duplication between the static and dynamic usage stats
  and adding the necessary state tracking to allow discriminating
  starting and started resources from one another
- drop `calculate_node_loads(...)` as it's currently unused
- drop `select_best_balancing_migration()` variants as these can be
  easily implemented with `score_best_balancing_migrations(..., 1)` in
  pve-ha-manager itself
- rename `score_best_balancing_migrations()` to
  `score_best_balancing_migration_candidates()` to make it possible to
  implement the former if we ever move the migration candidate
  generation to `Usage` or `Scheduler`

[0] https://lore.proxmox.com/pve-devel/DH6X1GK1YI3W.24PDWP36RSCVU@proxmox.com/




== Model ==

The automatic load rebalancing system checks whether the cluster node
imbalance exceeds some user-defined threshold for some HA Manager rounds
("hold duration"). If it does exceed on consecutive HA Manager rounds,
it will choose the best service migration/relocation to improve the
cluster node imbalance and queue it if it significantly improves it by
some user-defined improvement ("margin").

The best service motion can be selected by either bruteforce or TOPSIS.
This selection method and some other parameters from above can be
tweaked at runtime.



== Tests ==

I've added some test cases to ensure more basic decisions are
documented. The other tests were in virtualized clusters with adding
load dynamically to guests with stress-ng, even though I plan to rely
more on real-world load simulators for the next batch of tests.



== Benchmarks ==

I've also done some theoretical benchmarks with the target of being able
to handle a 48 nodes cluster and 9.999 HA resources / guests and a
worst-case scenario of each HA resource being part of 3 HA rules
(pairwise positive and negative resource affinity rules, where each
positive resource affinity pair has a common node affinity rule).

Generating the migration candidates for the huge cluster with the
worst-case HA ruleset takes 243 +- 9 ms.

Generating the migration candidates for the huge cluster without the
worst-case HA ruleset (to gain the most amount of 459954 migration
candidates) takes 356 +- 6 ms. This is expected, because we need to
evaluate more HA resources' rules as there are no HA resource bundles.

Excluding the generation, the brute force and TOPSIS method for
select_best_balancing_migration() were roughly similar both being in the
range 350 +- 50 ms for the huge cluster without any HA rules (for the
maximum amount of migration candidates) including the serialization
between Perl and Rust.



== Future ideas ==

- include the migration costs in score_best_balancing_migrations(),
  e.g., so that VMs with lots of memory are less likely to be migrated
  if the link between the nodes is slow, but that would need measuring
  and storing the migration network link speeds as a mesh

- apply some filter like moving average window or exponential smoothing
  on the usage time series to dampen spikes; triple exponential
  smoothing (Holts-Winters) is also already implemented in rrdcached and
  allows for exponential smoothing with better time series analysis but
  would require changing the rrdcached data structure once more

- score_best_balancing_migrations(...) can already provide a
  size-limited list of the best migrations, which could be exposed to
  users to allow manual load balancing actions, e.g., from the web
  interface, to get some insight in the system

- The current scheduler can only solve bin covering, but it would be
  interesting to also allow bin packing if certain criteria are met,
  e.g., for energy preservation while the overall cluster load is low

- Allow individual HA resources to be actively excluded from the
  automatic rebalancing, e.g., because containers cannot be live
  migrated.

- move the migration candidate generation to the rust-side; the
  generation on the perl-side was chosen first to reduce code
  duplication, but it doesn't seem future proof and right to copy state
  to the online_node_usage object twice (medium priority)



== Diffstat ==


proxmox:

Daniel Kral (9):
  resource-scheduling: inline add_cpu_usage in
    score_nodes_to_start_service
  resource-scheduling: move score_nodes_to_start_service to scheduler
    crate
  resource-scheduling: rename service to resource where appropriate
  resource-scheduling: introduce generic scheduler implementation
  resource-scheduling: implement generic cluster usage implementation
  resource-scheduling: topsis: handle empty criteria without panics
  resource-scheduling: compare by nodename in
    score_nodes_to_start_resource
  resource-scheduling: factor out topsis alternative mapping
  resource-scheduling: implement rebalancing migration selection

 proxmox-resource-scheduling/src/lib.rs        |   9 +
 proxmox-resource-scheduling/src/node.rs       |  96 +++++
 proxmox-resource-scheduling/src/pve_static.rs | 102 ++---
 proxmox-resource-scheduling/src/resource.rs   | 152 +++++++
 proxmox-resource-scheduling/src/scheduler.rs  | 406 ++++++++++++++++++
 proxmox-resource-scheduling/src/topsis.rs     |   6 +-
 proxmox-resource-scheduling/src/usage.rs      | 183 ++++++++
 .../tests/scheduler.rs                        | 367 ++++++++++++++++
 proxmox-resource-scheduling/tests/usage.rs    | 153 +++++++
 9 files changed, 1406 insertions(+), 68 deletions(-)
 create mode 100644 proxmox-resource-scheduling/src/node.rs
 create mode 100644 proxmox-resource-scheduling/src/resource.rs
 create mode 100644 proxmox-resource-scheduling/src/scheduler.rs
 create mode 100644 proxmox-resource-scheduling/src/usage.rs
 create mode 100644 proxmox-resource-scheduling/tests/scheduler.rs
 create mode 100644 proxmox-resource-scheduling/tests/usage.rs


perl-rs:

Daniel Kral (7):
  pve-rs: resource-scheduling: remove pedantic error handling from
    remove_node
  pve-rs: resource-scheduling: remove pedantic error handling from
    remove_service_usage
  pve-rs: resource-scheduling: move pve_static into resource_scheduling
    module
  pve-rs: resource-scheduling: use generic usage implementation
  pve-rs: resource-scheduling: static: replace deprecated usage structs
  pve-rs: resource-scheduling: implement pve_dynamic bindings
  pve-rs: resource-scheduling: expose auto rebalancing methods

 pve-rs/Makefile                               |   1 +
 pve-rs/src/bindings/mod.rs                    |   3 +-
 .../src/bindings/resource_scheduling/mod.rs   |  10 +
 .../resource_scheduling/pve_dynamic.rs        | 227 ++++++++++++++++++
 .../resource_scheduling/pve_static.rs         | 225 +++++++++++++++++
 .../bindings/resource_scheduling/resource.rs  | 128 ++++++++++
 .../src/bindings/resource_scheduling/usage.rs |  81 +++++++
 .../bindings/resource_scheduling_static.rs    | 215 -----------------
 pve-rs/test/resource_scheduling.pl            |   1 +
 9 files changed, 674 insertions(+), 217 deletions(-)
 create mode 100644 pve-rs/src/bindings/resource_scheduling/mod.rs
 create mode 100644 pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
 create mode 100644 pve-rs/src/bindings/resource_scheduling/pve_static.rs
 create mode 100644 pve-rs/src/bindings/resource_scheduling/resource.rs
 create mode 100644 pve-rs/src/bindings/resource_scheduling/usage.rs
 delete mode 100644 pve-rs/src/bindings/resource_scheduling_static.rs


cluster:

Daniel Kral (3):
  datacenter config: restructure verbose description for the ha crs
    option
  datacenter config: add dynamic load scheduler option
  datacenter config: add auto rebalancing options

 src/PVE/DataCenterConfig.pm | 51 ++++++++++++++++++++++++++++++++++---
 1 file changed, 47 insertions(+), 4 deletions(-)


ha-manager:

Daniel Kral (15):
  env: pve2: implement dynamic node and service stats
  usage: pass service data to add_service_usage
  usage: pass service data to get_used_service_nodes
  add running flag to cluster service stats
  usage: use add_service to add service usage to nodes
  usage: add dynamic usage scheduler
  test: add dynamic usage scheduler test cases
  manager: rename execute_migration to queue_resource_motion
  manager: update_crs_scheduler_mode: factor out crs config
  implement automatic rebalancing
  test: add resource bundle generation test cases
  test: add dynamic automatic rebalancing system test cases
  test: add static automatic rebalancing system test cases
  test: add automatic rebalancing system test cases with TOPSIS method
  test: add automatic rebalancing system test cases with affinity rules

Dominik Rusovac (6):
  sim: hardware: pass correct types for static stats
  sim: hardware: factor out static stats' default values
  sim: hardware: fix static stats guard
  sim: hardware: handle dynamic service stats
  sim: hardware: add set-dynamic-stats command
  sim: hardware: add getters for dynamic {node,service} stats

 debian/pve-ha-manager.install                 |   1 +
 src/PVE/HA/Env.pm                             |  12 +
 src/PVE/HA/Env/PVE2.pm                        |  64 +++++
 src/PVE/HA/Manager.pm                         | 218 +++++++++++++++-
 src/PVE/HA/Rules/ResourceAffinity.pm          |   3 +-
 src/PVE/HA/Sim/Env.pm                         |  12 +
 src/PVE/HA/Sim/Hardware.pm                    | 183 ++++++++++++--
 src/PVE/HA/Sim/RTHardware.pm                  |   4 +-
 src/PVE/HA/Usage.pm                           |  61 ++++-
 src/PVE/HA/Usage/Basic.pm                     |   9 +-
 src/PVE/HA/Usage/Dynamic.pm                   | 143 +++++++++++
 src/PVE/HA/Usage/Makefile                     |   2 +-
 src/PVE/HA/Usage/Static.pm                    |  53 +++-
 src/test/Makefile                             |   1 +
 .../README                                    |   2 +
 .../cmdlist                                   |   3 +
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   1 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  11 +
 .../manager_status                            |   1 +
 .../service_config                            |   1 +
 .../static_service_stats                      |   1 +
 .../README                                    |   7 +
 .../cmdlist                                   |   3 +
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   3 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  25 ++
 .../manager_status                            |   1 +
 .../service_config                            |   3 +
 .../static_service_stats                      |   3 +
 .../README                                    |   4 +
 .../cmdlist                                   |   3 +
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   6 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |   1 +
 .../service_config                            |   6 +
 .../static_service_stats                      |   6 +
 .../README                                    |   4 +
 .../cmdlist                                   |  16 ++
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   9 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  80 ++++++
 .../manager_status                            |   1 +
 .../service_config                            |   9 +
 .../static_service_stats                      |   9 +
 .../README                                    |  11 +
 .../cmdlist                                   |  13 +
 .../datacenter.cfg                            |   9 +
 .../dynamic_service_stats                     |   9 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |   1 +
 .../service_config                            |   9 +
 .../static_service_stats                      |   9 +
 .../test-crs-dynamic-auto-rebalance0/README   |   2 +
 .../test-crs-dynamic-auto-rebalance0/cmdlist  |   3 +
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   1 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  11 +
 .../manager_status                            |   1 +
 .../service_config                            |   1 +
 .../static_service_stats                      |   1 +
 .../test-crs-dynamic-auto-rebalance1/README   |   7 +
 .../test-crs-dynamic-auto-rebalance1/cmdlist  |   3 +
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   3 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  25 ++
 .../manager_status                            |   1 +
 .../service_config                            |   3 +
 .../static_service_stats                      |   3 +
 .../test-crs-dynamic-auto-rebalance2/README   |   4 +
 .../test-crs-dynamic-auto-rebalance2/cmdlist  |   3 +
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   6 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |   1 +
 .../service_config                            |   6 +
 .../static_service_stats                      |   6 +
 .../test-crs-dynamic-auto-rebalance3/README   |   4 +
 .../test-crs-dynamic-auto-rebalance3/cmdlist  |  16 ++
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   9 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  80 ++++++
 .../manager_status                            |   1 +
 .../service_config                            |   9 +
 .../static_service_stats                      |   9 +
 .../test-crs-dynamic-auto-rebalance4/README   |  11 +
 .../test-crs-dynamic-auto-rebalance4/cmdlist  |  13 +
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   9 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |   1 +
 .../service_config                            |   9 +
 .../static_service_stats                      |   9 +
 .../README                                    |   7 +
 .../cmdlist                                   |   8 +
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   5 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  49 ++++
 .../manager_status                            |   1 +
 .../rules_config                              |   4 +
 .../service_config                            |   5 +
 .../static_service_stats                      |   5 +
 .../README                                    |  12 +
 .../cmdlist                                   |   8 +
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   4 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  53 ++++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   4 +
 .../static_service_stats                      |   4 +
 .../README                                    |  14 ++
 .../cmdlist                                   |   3 +
 .../datacenter.cfg                            |   8 +
 .../dynamic_service_stats                     |   6 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |  31 +++
 .../rules_config                              |   3 +
 .../service_config                            |   6 +
 .../static_service_stats                      |   6 +
 .../README                                    |  14 ++
 .../cmdlist                                   |   3 +
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   6 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |   1 +
 .../rules_config                              |   7 +
 .../service_config                            |   6 +
 .../static_service_stats                      |   6 +
 src/test/test-crs-dynamic-rebalance1/README   |   3 +
 src/test/test-crs-dynamic-rebalance1/cmdlist  |   4 +
 .../datacenter.cfg                            |   7 +
 .../dynamic_service_stats                     |   7 +
 .../hardware_status                           |   5 +
 .../test-crs-dynamic-rebalance1/log.expect    |  88 +++++++
 .../manager_status                            |   1 +
 .../service_config                            |   7 +
 .../static_service_stats                      |   7 +
 src/test/test-crs-dynamic1/README             |   4 +
 src/test/test-crs-dynamic1/cmdlist            |   4 +
 src/test/test-crs-dynamic1/datacenter.cfg     |   6 +
 .../test-crs-dynamic1/dynamic_service_stats   |   3 +
 src/test/test-crs-dynamic1/hardware_status    |   5 +
 src/test/test-crs-dynamic1/log.expect         |  51 ++++
 src/test/test-crs-dynamic1/manager_status     |   1 +
 src/test/test-crs-dynamic1/service_config     |   3 +
 .../test-crs-dynamic1/static_service_stats    |   3 +
 .../test-crs-static-auto-rebalance1/README    |   7 +
 .../test-crs-static-auto-rebalance1/cmdlist   |   3 +
 .../datacenter.cfg                            |   7 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  25 ++
 .../manager_status                            |   1 +
 .../service_config                            |   3 +
 .../static_service_stats                      |   3 +
 .../test-crs-static-auto-rebalance2/README    |   4 +
 .../test-crs-static-auto-rebalance2/cmdlist   |   3 +
 .../datacenter.cfg                            |   7 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  59 +++++
 .../manager_status                            |   1 +
 .../service_config                            |   6 +
 .../static_service_stats                      |   6 +
 .../test-crs-static-auto-rebalance3/README    |   3 +
 .../test-crs-static-auto-rebalance3/cmdlist   |  15 ++
 .../datacenter.cfg                            |   7 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  79 ++++++
 .../manager_status                            |   1 +
 .../service_config                            |   9 +
 .../static_service_stats                      |   9 +
 src/test/test_resource_bundles.pl             | 234 ++++++++++++++++++
 187 files changed, 2791 insertions(+), 49 deletions(-)
 create mode 100644 src/PVE/HA/Usage/Dynamic.pm
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/README
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/cmdlist
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/hardware_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/log.expect
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/manager_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/rules_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/service_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/README
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/cmdlist
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/hardware_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/log.expect
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/manager_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/rules_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/service_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/README
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/cmdlist
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/hardware_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/log.expect
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/manager_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/rules_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/service_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/README
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/cmdlist
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/hardware_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/log.expect
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/manager_status
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/rules_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/service_config
 create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-rebalance1/README
 create mode 100644 src/test/test-crs-dynamic-rebalance1/cmdlist
 create mode 100644 src/test/test-crs-dynamic-rebalance1/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-rebalance1/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-rebalance1/hardware_status
 create mode 100644 src/test/test-crs-dynamic-rebalance1/log.expect
 create mode 100644 src/test/test-crs-dynamic-rebalance1/manager_status
 create mode 100644 src/test/test-crs-dynamic-rebalance1/service_config
 create mode 100644 src/test/test-crs-dynamic-rebalance1/static_service_stats
 create mode 100644 src/test/test-crs-dynamic1/README
 create mode 100644 src/test/test-crs-dynamic1/cmdlist
 create mode 100644 src/test/test-crs-dynamic1/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic1/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic1/hardware_status
 create mode 100644 src/test/test-crs-dynamic1/log.expect
 create mode 100644 src/test/test-crs-dynamic1/manager_status
 create mode 100644 src/test/test-crs-dynamic1/service_config
 create mode 100644 src/test/test-crs-dynamic1/static_service_stats
 create mode 100644 src/test/test-crs-static-auto-rebalance1/README
 create mode 100644 src/test/test-crs-static-auto-rebalance1/cmdlist
 create mode 100644 src/test/test-crs-static-auto-rebalance1/datacenter.cfg
 create mode 100644 src/test/test-crs-static-auto-rebalance1/hardware_status
 create mode 100644 src/test/test-crs-static-auto-rebalance1/log.expect
 create mode 100644 src/test/test-crs-static-auto-rebalance1/manager_status
 create mode 100644 src/test/test-crs-static-auto-rebalance1/service_config
 create mode 100644 src/test/test-crs-static-auto-rebalance1/static_service_stats
 create mode 100644 src/test/test-crs-static-auto-rebalance2/README
 create mode 100644 src/test/test-crs-static-auto-rebalance2/cmdlist
 create mode 100644 src/test/test-crs-static-auto-rebalance2/datacenter.cfg
 create mode 100644 src/test/test-crs-static-auto-rebalance2/hardware_status
 create mode 100644 src/test/test-crs-static-auto-rebalance2/log.expect
 create mode 100644 src/test/test-crs-static-auto-rebalance2/manager_status
 create mode 100644 src/test/test-crs-static-auto-rebalance2/service_config
 create mode 100644 src/test/test-crs-static-auto-rebalance2/static_service_stats
 create mode 100644 src/test/test-crs-static-auto-rebalance3/README
 create mode 100644 src/test/test-crs-static-auto-rebalance3/cmdlist
 create mode 100644 src/test/test-crs-static-auto-rebalance3/datacenter.cfg
 create mode 100644 src/test/test-crs-static-auto-rebalance3/hardware_status
 create mode 100644 src/test/test-crs-static-auto-rebalance3/log.expect
 create mode 100644 src/test/test-crs-static-auto-rebalance3/manager_status
 create mode 100644 src/test/test-crs-static-auto-rebalance3/service_config
 create mode 100644 src/test/test-crs-static-auto-rebalance3/static_service_stats
 create mode 100755 src/test/test_resource_bundles.pl


Summary over all repositories:
  206 files changed, 4918 insertions(+), 338 deletions(-)

-- 
Generated by murpp 0.11.0