From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH guest-common 01/15] scheduling: add cluster resource scheduling library
Date: Wed, 18 Nov 2026 12:05:08 +0100 [thread overview]
Message-ID: <20261118110705.595989-2-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118110705.595989-1-t.lamprecht@proxmox.com>
The CRS node-scheduling algorithms are currently only reachable from
inside the HA manager. Provide them as a library that both the HA stack
and plain API calls can build on, so ranking migration targets for a
guest or proposing a cluster rebalance does not need the HA manager,
and both sides share one implementation.
PVE::ResourceScheduling::Stats gathers the node and guest stats the HA
manager uses today, including the host memory accounting for VMs from
pve-ha-manager's fix #7974. Static guest usage comes from resolvers the
caller passes in, keeping the library free of guest type and HA
dependencies. PVE::ResourceScheduling::Scheduler is a plain forwarder
over the static and dynamic scheduler bindings, so the HA usage drivers
can build on it without any change in their decisions.
PVE::ResourceScheduling::Planner works on a snapshot of the whole
cluster. In dynamic mode it counts a node's load as at least the sum of
its running guests' usage, so a simulated migration removes from the
source what it adds to the target, while host overhead such as the ZFS
ARC stays counted. Nodes without usable stats are skipped instead of
failing the whole request, and the automatic mode only uses dynamic
stats if every node has them, so one node with stale data does not skew
the view of the others.
Rebalance proposals take the best migration step by step, for at most
ten steps by default, to keep a synchronous API call fast and the
proposal reviewable. A step must also improve the imbalance once
applied, as the static backend mispredicts migrating a container
without CPU limit. Otherwise it is undone, and the step tries the next
best guest instead. Steps gaining next to nothing, like moving an idle
guest, or little compared to the earlier ones, and proposals barely
improving an already balanced cluster, are dropped, as every migration
disrupts its guest. The result lists one net migration per guest,
since callers carry them out concurrently.
Guests that must be kept apart are checked against the placement
reached by the earlier steps, as per-guest target lists alone cannot
keep apart guests that move in the same proposal.
To keep a synchronous API call fast on large clusters, where scoring
every guest against every node takes about a second per step with 100
nodes and 10000 guests, a step only considers the guests on the
busiest nodes, each with the least loaded node of every node size.
Depend on libpve-rs-perl 0.14.0 or newer, the first version reporting
the imbalance normalized to the unit interval, which the improvement
thresholds are expressed in.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
debian/control | 2 +
src/Makefile | 4 +
src/PVE/ResourceScheduling/Planner.pm | 716 +++++++
src/PVE/ResourceScheduling/Scheduler.pm | 164 ++
src/PVE/ResourceScheduling/Stats.pm | 194 ++
src/tests/Makefile | 21 +-
.../resource-scheduling-planner-tests.pl | 1775 +++++++++++++++++
.../resource-scheduling-scheduler-tests.pl | 230 +++
src/tests/resource-scheduling-stats-tests.pl | 260 +++
9 files changed, 3364 insertions(+), 2 deletions(-)
create mode 100644 src/PVE/ResourceScheduling/Planner.pm
create mode 100644 src/PVE/ResourceScheduling/Scheduler.pm
create mode 100644 src/PVE/ResourceScheduling/Stats.pm
create mode 100644 src/tests/resource-scheduling-planner-tests.pl
create mode 100644 src/tests/resource-scheduling-scheduler-tests.pl
create mode 100644 src/tests/resource-scheduling-stats-tests.pl
diff --git a/debian/control b/debian/control
index 0adb2b6..de35dfe 100644
--- a/debian/control
+++ b/debian/control
@@ -5,6 +5,7 @@ Maintainer: Proxmox Support Team <support@proxmox.com>
Build-Depends: debhelper-compat (= 13),
libpve-cluster-perl,
libpve-common-perl (>= 9.2.6),
+ libpve-rs-perl (>= 0.14.0),
libpve-storage-perl (>= 8.3.4),
libtest-mockmodule-perl,
pve-cluster (>= 9.1.10),
@@ -16,6 +17,7 @@ Architecture: all
Depends: libpve-access-control,
libpve-cluster-perl (>= 8.1.0),
libpve-common-perl (>= 9.2.6),
+ libpve-rs-perl (>= 0.14.0),
libpve-storage-perl (>= 8.3.4),
proxmox-websocket-tunnel,
pve-cluster (>= 9.1.10),
diff --git a/src/Makefile b/src/Makefile
index 161dbe5..f44009b 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -19,6 +19,10 @@ install: PVE
install -m 0644 PVE/Mapping/Dir.pm ${PERL5DIR}/PVE/Mapping/
install -m 0644 PVE/Mapping/PCI.pm ${PERL5DIR}/PVE/Mapping/
install -m 0644 PVE/Mapping/USB.pm ${PERL5DIR}/PVE/Mapping/
+ install -d ${PERL5DIR}/PVE/ResourceScheduling
+ install -m 0644 PVE/ResourceScheduling/Planner.pm ${PERL5DIR}/PVE/ResourceScheduling/
+ install -m 0644 PVE/ResourceScheduling/Scheduler.pm ${PERL5DIR}/PVE/ResourceScheduling/
+ install -m 0644 PVE/ResourceScheduling/Stats.pm ${PERL5DIR}/PVE/ResourceScheduling/
install -d ${PERL5DIR}/PVE/VZDump
install -m 0644 PVE/VZDump/Common.pm ${PERL5DIR}/PVE/VZDump/
install -m 0644 PVE/VZDump/JobBase.pm ${PERL5DIR}/PVE/VZDump/
diff --git a/src/PVE/ResourceScheduling/Planner.pm b/src/PVE/ResourceScheduling/Planner.pm
new file mode 100644
index 0000000..c557cbc
--- /dev/null
+++ b/src/PVE/ResourceScheduling/Planner.pm
@@ -0,0 +1,716 @@
+package PVE::ResourceScheduling::Planner;
+
+use v5.36;
+
+use List::Util qw(max min);
+use Scalar::Util qw(looks_like_number);
+
+use PVE::ResourceScheduling::Scheduler;
+use PVE::ResourceScheduling::Stats;
+
+# A cluster-wide view of node and guest usage that drives the resource scheduler for two queries:
+# ranking the best node(s) to place or migrate a single guest, and proposing a set of migrations to
+# rebalance the cluster.
+#
+# This is the HA-independent counterpart to the HA stack's PVE::HA::Usage::* drivers; it works on
+# plain guest stats (node, running, usage) instead of HA service state, so it can be used from a
+# normal API call without the HA manager.
+
+=head3 DEFAULT_MAX_MOVES
+
+The number of greedy steps C<suggest_rebalance()> takes at most unless told otherwise. Every step
+scores movable guests against target nodes, so the cost of a proposal grows with its length; the
+default keeps a synchronous API call fast and the proposal short enough to review. Applying a
+proposal and asking again continues from the new state.
+
+=cut
+
+use constant DEFAULT_MAX_MOVES => 10;
+
+=head3 DEFAULT_BUSIEST_NODES
+
+The default C<busiest_nodes> of C<suggest_rebalance()>, 10. Scoring every movable guest against
+every other node makes a step expensive on large clusters, as the number of pairs grows with both
+the guests and the nodes, while the best migration practically always moves load off one of the
+busiest nodes. Limiting a step to the guests on those keeps a synchronous API call fast.
+
+=cut
+
+use constant DEFAULT_BUSIEST_NODES => 10;
+
+=head3 DEFAULT_MIN_STEP_RATIO
+
+The default C<min_step_ratio> of C<suggest_rebalance()>, 0.1. A greedy proposal takes the largest
+improvements first, so once a step gains less than a tenth of the best step taken, the remaining
+steps mostly shuffle guests around for a marginal gain, while each migration disrupts its guest, and
+a container even gets restarted.
+
+=cut
+
+use constant DEFAULT_MIN_STEP_RATIO => 0.1;
+
+=head3 DEFAULT_BALANCED_IMBALANCE
+
+The default C<balanced_imbalance> of C<suggest_rebalance()>, 0.1. Below that imbalance, moves are
+only proposed if together they lower it by at least C<min_total_ratio> of its value. Only such
+clusters need that guard: on a clearly imbalanced cluster with many small guests, the few steps
+of one proposal may not gain much in total, but each of them is a real improvement.
+
+=cut
+
+use constant DEFAULT_BALANCED_IMBALANCE => 0.1;
+
+=head3 DEFAULT_MIN_TOTAL_RATIO
+
+The default C<min_total_ratio> of C<suggest_rebalance()>, 0.1. For a cluster that counts as
+balanced, a proposal lowering the imbalance by less than this fraction of its value is not worth
+the disruption of its migrations. The threshold is relative, as the imbalance caused by a single
+overloaded node shrinks with the number of nodes, so an absolute one would hide such a node in a
+larger cluster.
+
+=cut
+
+use constant DEFAULT_MIN_TOTAL_RATIO => 0.1;
+
+# The scorer truncates the imbalance it predicts for a migration to 15 significant digits, so a move
+# that leaves the load distribution unchanged can look like an improvement of about 1e-16. Requiring
+# at least this much improvement per step keeps such no-op migrations out of a proposal.
+use constant MIN_IMPROVEMENT_FLOOR => 1e-9;
+
+# Evening out two nodes lowers the imbalance by about its value divided by the node count. A step
+# gaining less than this fraction of that, like the move of an idle guest, changes nothing
+# noticeable and is not worth a migration.
+use constant MIN_STEP_NODE_SHARE => 0.05;
+
+# How many moves a step may undo before it ends the proposal. Each retry rescores all candidates.
+use constant MAX_UNDONE_PER_STEP => 5;
+
+# Whether the scheduler can work with a node's stats. The backend divides by the capacity, so a
+# missing or zero capacity would turn every imbalance value into NaN.
+my sub usable_node_stats($stats) {
+ return
+ ref($stats) eq 'HASH'
+ && ($stats->{maxcpu} // 0) > 0
+ && ($stats->{maxmem} // 0) > 0;
+}
+
+# Copy the stats the given mode uses, with absent dynamic load counted as idle.
+my sub copy_node_stats($mode, $stats) {
+ my $copy = { maxcpu => $stats->{maxcpu}, maxmem => $stats->{maxmem} };
+ if ($mode eq 'dynamic') {
+ $copy->{cpu} = $stats->{cpu} // 0;
+ $copy->{mem} = $stats->{mem} // 0;
+ }
+ return $copy;
+}
+
+my sub copy_guest_stats($guest_stats) {
+ my $copy = {};
+ for my $sid (keys %$guest_stats) {
+ my $guest = $guest_stats->{$sid};
+ $copy->{$sid} = { $guest->%* };
+ $copy->{$sid}->{usage} = { $guest->{usage}->%* } if defined($guest->{usage});
+ }
+ return $copy;
+}
+
+# The running guests on each of the usable nodes in $node_stats, as { $node => { $sid => 1 } }. Only
+# these put load on a node; guests on other nodes are treated like guests on an offline node.
+my sub running_guests_by_node($node_stats, $guest_stats) {
+ my $by_node = { map { $_ => {} } keys %$node_stats };
+ for my $sid (keys %$guest_stats) {
+ my $guest = $guest_stats->{$sid};
+ next if !$guest->{running};
+
+ my $node = $guest->{node};
+ next if !defined($node) || !$node_stats->{$node};
+
+ $by_node->{$node}->{$sid} = 1;
+ }
+
+ return $by_node;
+}
+
+# Add $node with its running guests to $scheduler. Returns the node's load as the scheduler sees it,
+# as fractions of its capacity: { cpu, mem }.
+my sub add_node_with_guests($scheduler, $mode, $node, $stats, $sids, $guest_stats) {
+ my @sids = sort keys %$sids;
+
+ my ($cpu, $mem) = (0, 0);
+ for my $sid (@sids) {
+ my $usage = $guest_stats->{$sid}->{usage} // {};
+ if ($mode eq 'static') {
+ # the static backend counts a guest without CPU limit with all cores of its node
+ $cpu += ($usage->{maxcpu} || $stats->{maxcpu});
+ $mem += $usage->{maxmem} // 0;
+ } else {
+ $cpu += $usage->{cpu} // 0;
+ $mem += $usage->{mem} // 0;
+ }
+ }
+
+ if ($mode eq 'static') {
+ $scheduler->add_node($node, $stats);
+ } else {
+ # Reconcile node and guest stats: host overhead not attributable to any guest (e.g. ZFS
+ # ARC, page cache) stays counted as a fixed per-node base, but the node load is never
+ # reported below the sum of its running guests, so removing a guest in a simulated
+ # migration cannot drive a node below the load of its remaining guests. Only the planner
+ # works on a snapshot, so this lives here and not in the scheduler, which stays a pure
+ # forwarder for the HA stack.
+ $cpu = max($stats->{cpu}, $cpu);
+ $mem = max($stats->{mem}, $mem);
+ $scheduler->add_node(
+ $node,
+ {
+ maxcpu => $stats->{maxcpu},
+ cpu => $cpu,
+ maxmem => $stats->{maxmem},
+ mem => $mem,
+ },
+ );
+ }
+
+ for my $sid (@sids) {
+ # always add guests as stationary on their current node: this is a planning view of the
+ # committed placement, and the scheduler only accepts stationary migration candidates
+ $scheduler->add_resource(
+ $sid,
+ { stats => $guest_stats->{$sid}->{usage}, running => 1, 'current-node' => $node },
+ );
+ }
+
+ return { cpu => $cpu / $stats->{maxcpu}, mem => $mem / $stats->{maxmem} };
+}
+
+# Build and populate a fresh scheduler from a snapshot whose $node_stats hold exactly the usable
+# nodes. Returns the scheduler, the running guests by node and the node loads, see
+# running_guests_by_node() and add_node_with_guests().
+my sub build_scheduler($mode, $node_stats, $guest_stats) {
+ my $scheduler = PVE::ResourceScheduling::Scheduler->new($mode);
+ my $by_node = running_guests_by_node($node_stats, $guest_stats);
+
+ my $loads = {};
+ for my $node (sort keys %$node_stats) {
+ $loads->{$node} = add_node_with_guests(
+ $scheduler, $mode, $node, $node_stats->{$node}, $by_node->{$node}, $guest_stats,
+ );
+ }
+
+ return ($scheduler, $by_node, $loads);
+}
+
+# Move guest $sid from $source to $target in the working state of a rebalance proposal and update
+# the scheduler. Only the two nodes involved are re-added, which is much cheaper than a rebuild on
+# large clusters. In dynamic mode the guest's measured usage moves with it, matching how the
+# scheduler predicts a migration; a reverted move restores the saved measured load instead.
+my sub move_guest($work, $sid, $source, $target, $saved = undef) {
+ my ($mode, $node_stats, $guest_stats) = $work->@{qw(mode node_stats guest_stats)};
+
+ if ($saved) {
+ $node_stats->@{ keys %$saved } = values %$saved;
+ } elsif ($mode eq 'dynamic') {
+ my $usage = $guest_stats->{$sid}->{usage} // {};
+ my ($from, $to) = $node_stats->@{ $source, $target };
+
+ $from->{cpu} = max(0, $from->{cpu} - ($usage->{cpu} // 0));
+ $from->{mem} = max(0, $from->{mem} - ($usage->{mem} // 0));
+ $to->{cpu} += $usage->{cpu} // 0;
+ $to->{mem} += $usage->{mem} // 0;
+ }
+
+ $guest_stats->{$sid}->{node} = $target;
+ for my $index ($work->@{qw(by_node movable_by_node)}) {
+ delete $index->{$source}->{$sid};
+ $index->{$target}->{$sid} = 1;
+ }
+
+ for my $node ($source, $target) {
+ $work->{scheduler}->remove_node($node);
+ $work->{loads}->{$node} = add_node_with_guests(
+ $work->{scheduler},
+ $mode,
+ $node,
+ $node_stats->{$node},
+ $work->{by_node}->{$node},
+ $guest_stats,
+ );
+ }
+
+ return;
+}
+
+# The load of $node as the scheduler sees it, as a fraction of its capacity: per dimension ('cpu' or
+# 'mem') or combined ('load'), the latter being what the imbalance is computed from.
+my sub node_load($loads, $node, $dim) {
+ my $load = $loads->{$node};
+ return $dim eq 'load' ? ($load->{cpu} + $load->{mem}) / 2 : $load->{$dim};
+}
+
+# The migration candidates for one step of a rebalance proposal: each movable guest with the target
+# nodes it may go to. Scoring all pairs of guests and nodes is what makes a step expensive on large
+# clusters, so with $limit, only the guests on the $limit most loaded nodes per load dimension are
+# considered, as the best migration practically always moves load off a busy node.
+#
+# Their targets are pruned without losing a better move: a guest adds the same fraction of capacity
+# to any node of the same size, that is with the same CPU and memory capacity, so among these the
+# one with the least combined load is the target with the lowest resulting imbalance. Only nodes of
+# different size compete, so each guest is offered the least loaded node of every size.
+my sub rebalance_candidates($work, $limit) {
+ my ($loads, $movable_by_node, $allowed) = $work->@{qw(loads movable_by_node allowed)};
+
+ my @sources = grep { $movable_by_node->{$_}->%* } sort keys %$movable_by_node;
+ my $orders;
+ if ($limit) {
+ my $pick = {};
+ for my $dim (qw(load cpu mem)) {
+ my @busiest = sort {
+ node_load($loads, $b, $dim) <=> node_load($loads, $a, $dim) || $a cmp $b
+ } @sources;
+ $pick->{$_} = 1 for @busiest[0 .. min($limit, scalar(@busiest)) - 1];
+ }
+ @sources = sort keys %$pick;
+
+ my $sizes = {};
+ for my $node ($work->{target_nodes}->@*) {
+ my $stats = $work->{node_stats}->{$node};
+ push $sizes->{"$stats->{maxcpu}/$stats->{maxmem}"}->@*, $node;
+ }
+
+ $orders = [];
+ for my $size (sort keys %$sizes) {
+ push @$orders, [
+ sort {
+ node_load($loads, $a, 'load') <=> node_load($loads, $b, 'load') || $a cmp $b
+ } $sizes->{$size}->@*
+ ];
+ }
+ }
+
+ my $targets_of = sub($sid, $source) {
+ my $ok = $allowed->{$sid};
+
+ # never its current node, nor one its partners are on after the earlier steps
+ my $forbidden = { $source => 1 };
+ for my $partner (keys(($work->{partners}->{$sid} // {})->%*)) {
+ my $node = $work->{guest_stats}->{$partner}->{node};
+ $forbidden->{$node} = 1 if defined($node);
+ }
+
+ if (!$orders) {
+ return [grep { !$forbidden->{$_} && (!$ok || $ok->{$_}) }
+ $work->{target_nodes}->@*];
+ }
+ my $pick = {};
+ for my $order (@$orders) {
+ for my $node (@$order) {
+ next if $forbidden->{$node} || ($ok && !$ok->{$node});
+ $pick->{$node} = 1;
+ last;
+ }
+ }
+ return [sort keys %$pick];
+ };
+
+ my @candidates;
+ for my $source (@sources) {
+ my $shared; # target nodes shared by all guests without a restriction of their own
+ for my $sid (sort keys $movable_by_node->{$source}->%*) {
+ my $nodes =
+ $allowed->{$sid} || $work->{partners}->{$sid}
+ ? $targets_of->($sid, $source)
+ : ($shared //= $targets_of->($sid, $source));
+ push @candidates, { leader => $sid, resources => [$sid], nodes => $nodes } if @$nodes;
+ }
+ }
+
+ return [sort { $a->{leader} cmp $b->{leader} } @candidates];
+}
+
+=head3 PVE::ResourceScheduling::Planner->new(%args)
+
+Construct from an explicit stats snapshot given in C<%args>: C<mode> ('dynamic' (default) or
+'static'), C<nodes> (arrayref of node names), C<node_stats> (C<< { $node => { maxcpu, maxmem, cpu?,
+mem? } } >>) and C<guest_stats> (C<< { $sid => { node, running, usage } } >>, as returned by
+C<PVE::ResourceScheduling::Stats>). The snapshot is copied, so later changes to the passed hashes
+do not affect the planner.
+
+Only the nodes from C<nodes> with usable stats take part, that is a stats hash with C<maxcpu> and
+C<maxmem> greater than zero; in dynamic mode a missing C<cpu> or C<mem> counts as idle. The other
+nodes are skipped: they are neither ranking candidates nor rebalance sources or targets, and their
+guests are treated like guests on an offline node. Mostly for tests and internal use; production
+callers usually want C<build()>.
+
+=cut
+
+sub new($class, %args) {
+ my $mode = $args{mode} // 'dynamic';
+ die "invalid scheduling mode '$mode'\n" if $mode ne 'dynamic' && $mode ne 'static';
+
+ my $nodes = $args{nodes} or die "missing required 'nodes' argument\n";
+ my $node_stats = $args{node_stats} or die "missing required 'node_stats' argument\n";
+ my $guest_stats = $args{guest_stats} // {};
+
+ my $usable = {};
+ for my $node (@$nodes) {
+ my $stats = $node_stats->{$node};
+ $usable->{$node} = copy_node_stats($mode, $stats) if usable_node_stats($stats);
+ }
+
+ my $self = bless {
+ mode => $mode,
+ node_stats => $usable,
+ guest_stats => copy_guest_stats($guest_stats),
+ }, $class;
+
+ ($self->{scheduler}) = build_scheduler($mode, $self->{node_stats}, $self->{guest_stats});
+
+ return $self;
+}
+
+=head3 PVE::ResourceScheduling::Planner->build(%args)
+
+Gather the cluster stats and construct. C<%args> takes C<nodes> (arrayref of node names, required),
+C<mode> ('auto' (default), 'dynamic' or 'static') and C<resolvers> (a guest-type to code-ref map,
+see C<PVE::ResourceScheduling::Stats::get_static_guest_stats>), which is required for 'auto' and
+'static'.
+
+'auto' uses dynamic load if every requested node has usable dynamic stats, and static usage
+otherwise, so one node without fresh load data does not distort the view of the others. An explicit
+'dynamic' or 'static' uses that mode and skips the nodes without usable stats. Errors while
+gathering the stats are not caught.
+
+=cut
+
+sub build($class, %args) {
+ my $mode = $args{mode} // 'auto';
+ die "invalid scheduling mode '$mode'\n"
+ if $mode ne 'auto' && $mode ne 'dynamic' && $mode ne 'static';
+
+ my $nodes = $args{nodes} or die "missing required 'nodes' argument\n";
+
+ my $resolvers = $args{resolvers};
+ die "scheduling mode '$mode' needs a 'resolvers' argument\n"
+ if $mode ne 'dynamic' && ref($resolvers) ne 'HASH';
+
+ if ($mode ne 'static') {
+ my $node_stats = PVE::ResourceScheduling::Stats::get_dynamic_node_stats();
+
+ my $complete = !grep { !usable_node_stats($node_stats->{$_}) } @$nodes;
+ if ($mode eq 'dynamic' || $complete) {
+ return $class->new(
+ mode => 'dynamic',
+ nodes => $nodes,
+ node_stats => $node_stats,
+ guest_stats => PVE::ResourceScheduling::Stats::get_dynamic_guest_stats(),
+ );
+ }
+ }
+
+ return $class->new(
+ mode => 'static',
+ nodes => $nodes,
+ node_stats => PVE::ResourceScheduling::Stats::get_static_node_stats(),
+ guest_stats => PVE::ResourceScheduling::Stats::get_static_guest_stats($resolvers),
+ );
+}
+
+=head3 $planner->mode()
+
+The scheduling mode actually in use, 'dynamic' or 'static'.
+
+=cut
+
+sub mode($self) {
+ return $self->{mode};
+}
+
+=head3 $planner->node_imbalance()
+
+The current cluster load imbalance in C<[0, 1]>; lower is more balanced.
+
+=cut
+
+sub node_imbalance($self) {
+ return $self->{scheduler}->calculate_node_imbalance();
+}
+
+=head3 $planner->rank_target_nodes($sid, $candidate_nodes)
+
+Rank the target nodes for guest C<$sid> of the snapshot, best first, as an arrayref of
+C<< { node, imbalance? } >>. The candidates are the snapshot nodes, restricted to the optional
+C<$candidate_nodes> arrayref, never including the guest's current node.
+
+A guest running on a snapshot node is ranked as a migration off that node, and each entry carries
+the resulting cluster C<imbalance> (lower is better). Any other guest, stopped or on a skipped or
+offline node, is ranked as a placement by its usage, and the entries carry no C<imbalance>. Dies if
+C<$sid> is not in the snapshot.
+
+=cut
+
+sub rank_target_nodes($self, $sid, $candidate_nodes = undef) {
+ my $guest = $self->{guest_stats}->{$sid}
+ or die "no such guest '$sid' in the cluster snapshot\n";
+
+ my $nodes = $self->{node_stats};
+ my $current = $guest->{node} // '';
+ my $wanted = $candidate_nodes ? { map { $_ => 1 } @$candidate_nodes } : undef;
+
+ my @candidates = grep { $_ ne $current && (!$wanted || $wanted->{$_}) } sort keys %$nodes;
+ return [] if !@candidates;
+
+ if ($guest->{running} && $nodes->{$current}) {
+ my $candidate = { leader => $sid, resources => [$sid], nodes => [@candidates] };
+ my $scored = $self->{scheduler}
+ ->score_best_balancing_migration_candidates([$candidate], scalar(@candidates));
+
+ my @ranked =
+ map { { node => $_->{migration}->{'target-node'}, imbalance => $_->{imbalance} } }
+ @$scored;
+
+ # lower resulting imbalance is better, with the node name as a stable tie-break
+ return [sort { $a->{imbalance} <=> $b->{imbalance} || $a->{node} cmp $b->{node} } @ranked];
+ }
+
+ my $usage = $guest->{usage} or die "missing usage of guest '$sid' for placement ranking\n";
+ my $allowed = { map { $_ => 1 } @candidates };
+
+ # higher start score is better; the score itself is not exposed, only the resulting order
+ my @ranked =
+ sort { $b->[1] <=> $a->[1] || $a->[0] cmp $b->[0] }
+ grep { $allowed->{ $_->[0] } } $self->{scheduler}->score_nodes_to_start($usage)->@*;
+
+ return [map { { node => $_->[0] } } @ranked];
+}
+
+=head3 $planner->suggest_rebalance($movable, %opts)
+
+Propose migrations that reduce the cluster imbalance, moving only the guests in the C<$movable>
+arrayref of sids. Only guests running on a snapshot node are moved, and only between snapshot nodes.
+
+The optional C<$opts{targets}> arrayref of node names restricts the nodes guests may be moved to,
+for example to fill newly added nodes or to keep guests off a node. Names that are not snapshot
+nodes are ignored, and without any remaining target node no moves are proposed. Guests are still
+moved off any snapshot node, including one outside the targets, and the imbalance is still
+computed over all snapshot nodes.
+
+The optional C<$opts{allowed_targets}> hash maps a sid to an arrayref of the node names that guest
+may be moved to, for example to honor the node rules of an HA resource. For a guest in it, these
+nodes further restrict C<$opts{targets}>; a guest without remaining target node is not moved. Guests
+not in it are not restricted further.
+
+The optional C<$opts{separate}> hash maps a sid to an arrayref of sids of the guests it must be
+kept apart from, for example for a negative resource affinity rule; the relation is mutual, so
+listing it in one direction suffices. A step never moves a guest onto a node that, in the placement
+reached by the earlier steps, holds one of its partners, whether these are movable or not. So no
+reported move puts guests that must be kept apart on one node, while guests that already share a
+node are left for the caller to resolve.
+
+With the default method, each step considers only the guests on the C<$opts{busiest_nodes}>
+(default C<DEFAULT_BUSIEST_NODES>, 0 disables) most loaded nodes by combined, CPU and memory load,
+and offers each guest only its least loaded target node of every node size, which cannot miss a
+better move for these guests. If such a step finds no move passing the checks below, it is retried
+with all guests and target nodes, so the limit never ends a proposal early.
+
+Works greedily on a copy of the snapshot: each step picks the best migration according to
+C<$opts{method}>, 'bruteforce' (default, lowest resulting imbalance) or 'topsis'. The proposal ends
+after at most C<$opts{max_moves}> steps (default C<DEFAULT_MAX_MOVES>), or before the first step
+that falls short, that is, whose improvement of the imbalance
+
+=over
+
+=item * is not larger than C<$opts{min_improvement}> (default 0), or than a small floor that keeps
+no-op migrations out, or
+
+=item * is less than a twentieth of the current imbalance divided by the node count, which keeps
+moves of nearly idle guests out, or
+
+=item * is less than C<$opts{min_step_ratio}> (default C<DEFAULT_MIN_STEP_RATIO>, 0 disables) times
+the largest improvement of a step taken so far.
+
+=back
+
+The improvement is checked both as predicted by the scheduler and as measured on the state after
+the move. If the measured improvement falls short, the move is undone and the step tries the next
+best guest instead, up to a few times. With 'topsis', the proposal thus ends as soon as the
+method's best pick is not predicted to improve the imbalance enough.
+
+Below an imbalance of C<$opts{balanced_imbalance}> (default C<DEFAULT_BALANCED_IMBALANCE>, 0
+disables), moves are only proposed if together they lower the imbalance by at least
+C<$opts{min_total_ratio}> (default C<DEFAULT_MIN_TOTAL_RATIO>, 0 disables) of its value.
+
+Returns C<< { mode, imbalance => { before, after }, moves => [ { sid, source, target } ] } >>.
+C<after> is the imbalance once all moves are done. A guest moved more than once is reported with a
+single move from its original to its final node, and one that ends up on its original node is left
+out. As every step moves a guest to one of its target nodes, so does every reported move. The moves
+are ordered by the step in which each guest first moved. As each guest appears at most once,
+carrying out the moves in any order, or concurrently, leads to the same placement.
+
+=cut
+
+sub suggest_rebalance($self, $movable, %opts) {
+ my $method = $opts{method} // 'bruteforce';
+ die "invalid method '$method'\n" if $method ne 'bruteforce' && $method ne 'topsis';
+
+ my $max_moves = $opts{max_moves} // DEFAULT_MAX_MOVES;
+ die "invalid max_moves '$max_moves', must be a positive integer\n"
+ if $max_moves !~ m/^[0-9]+$/ || $max_moves < 1;
+
+ my $min_improvement = $opts{min_improvement} // 0;
+ die "invalid min_improvement '$min_improvement', must be a non-negative number\n"
+ if !looks_like_number($min_improvement) || !($min_improvement >= 0);
+
+ my $min_step_ratio = $opts{min_step_ratio} // DEFAULT_MIN_STEP_RATIO;
+ die "invalid min_step_ratio '$min_step_ratio', must be a number from 0 to 1\n"
+ if !looks_like_number($min_step_ratio) || !($min_step_ratio >= 0 && $min_step_ratio <= 1);
+
+ my $min_total_ratio = $opts{min_total_ratio} // DEFAULT_MIN_TOTAL_RATIO;
+ die "invalid min_total_ratio '$min_total_ratio', must be a number from 0 to 1\n"
+ if !looks_like_number($min_total_ratio)
+ || !($min_total_ratio >= 0 && $min_total_ratio <= 1);
+
+ my $balanced_imbalance = $opts{balanced_imbalance} // DEFAULT_BALANCED_IMBALANCE;
+ die "invalid balanced_imbalance '$balanced_imbalance', must be a number from 0 to 1\n"
+ if !looks_like_number($balanced_imbalance)
+ || !($balanced_imbalance >= 0 && $balanced_imbalance <= 1);
+
+ my $targets = $opts{targets};
+ die "invalid targets, must be an array reference of node names\n"
+ if defined($targets) && ref($targets) ne 'ARRAY';
+
+ my $allowed_targets = $opts{allowed_targets} // {};
+ die "invalid allowed_targets, must be a hash reference of node name array references\n"
+ if ref($allowed_targets) ne 'HASH' || grep { ref($_) ne 'ARRAY' } values %$allowed_targets;
+
+ my $separate = $opts{separate} // {};
+ die "invalid separate, must be a hash reference of guest id array references\n"
+ if ref($separate) ne 'HASH' || grep { ref($_) ne 'ARRAY' } values %$separate;
+
+ my $busiest_nodes = $opts{busiest_nodes} // DEFAULT_BUSIEST_NODES;
+ die "invalid busiest_nodes '$busiest_nodes', must be a non-negative integer\n"
+ if $busiest_nodes !~ m/^[0-9]+$/;
+
+ my $mode = $self->{mode};
+ my $work = {
+ mode => $mode,
+ node_stats =>
+ { map { $_ => { $self->{node_stats}->{$_}->%* } } keys $self->{node_stats}->%* },
+ guest_stats => copy_guest_stats($self->{guest_stats}),
+ };
+ ($work->{scheduler}, $work->{by_node}, $work->{loads}) =
+ build_scheduler($mode, $work->{node_stats}, $work->{guest_stats});
+
+ my $wanted = $targets ? { map { $_ => 1 } @$targets } : undef;
+ $work->{target_nodes} = [grep { !$wanted || $wanted->{$_} } sort keys $work->{node_stats}->%*];
+
+ for my $sid (keys %$allowed_targets) {
+ $work->{allowed}->{$sid} = { map { $_ => 1 } $allowed_targets->{$sid}->@* };
+ }
+
+ # keeping guests apart is mutual, whichever of them the caller listed
+ for my $sid (keys %$separate) {
+ for my $partner ($separate->{$sid}->@*) {
+ next if $partner eq $sid;
+ $work->{partners}->{$sid}->{$partner} = 1;
+ $work->{partners}->{$partner}->{$sid} = 1;
+ }
+ }
+
+ my $movable_set = { map { $_ => 1 } @$movable };
+ for my $node (keys $work->{by_node}->%*) {
+ $work->{movable_by_node}->{$node} =
+ { map { $_ => 1 } grep { $movable_set->{$_} } keys $work->{by_node}->{$node}->%* };
+ }
+
+ # Pruning only pays off and only makes a difference with more nodes than its limit. TOPSIS ranks
+ # the candidates relative to each other, so leaving some out could change its pick.
+ my $prune =
+ $method eq 'bruteforce'
+ && $busiest_nodes
+ && keys($work->{node_stats}->%*) > $busiest_nodes;
+
+ my $before = $work->{scheduler}->calculate_node_imbalance();
+ my $current = $before;
+
+ my $threshold = max($min_improvement, MIN_IMPROVEMENT_FLOOR);
+ my $node_count = keys $work->{node_stats}->%*;
+ my $largest_gain = 0; # of the steps taken so far
+
+ my $falls_short = sub($gain) {
+ return
+ $gain <= $threshold
+ || $gain < MIN_STEP_NODE_SHARE * $current / $node_count
+ || $gain < $min_step_ratio * $largest_gain;
+ };
+
+ my ($origin, $first_step) = ({}, {});
+ for my $step (1 .. $max_moves) {
+ my $taken;
+ my $undone = {}; # guests whose move this step undid
+
+ # A pruned step can miss the best move, so when it finds nothing worthwhile, retry the
+ # step with all candidates before ending the proposal.
+ LIMIT: for my $limit ($prune ? ($busiest_nodes, 0) : (0)) {
+ my $candidates = rebalance_candidates($work, $limit);
+
+ while (keys %$undone < MAX_UNDONE_PER_STEP) {
+ my @left = grep { !$undone->{ $_->{leader} } } @$candidates;
+ last if !@left;
+
+ my $best = $work->{scheduler}
+ ->score_best_balancing_migration_candidates(\@left, 1, $method)->[0];
+ last if !$best || $falls_short->($current - $best->{imbalance});
+
+ my ($sid, $source, $target) =
+ $best->{migration}->@{qw(sid source-node target-node)};
+ my $saved = { map { $_ => { $work->{node_stats}->{$_}->%* } } $source, $target };
+
+ move_guest($work, $sid, $source, $target);
+ my $actual = $work->{scheduler}->calculate_node_imbalance();
+
+ # The prediction can be off where the backend counts a guest differently in the
+ # node load than in a migration, e.g. a static-mode container without a CPU limit.
+ # Trust the updated state, undo a move that does not actually pay off and try the
+ # next best guest.
+ if ($falls_short->($current - $actual)) {
+ move_guest($work, $sid, $target, $source, $saved);
+ $undone->{$sid} = 1;
+ next;
+ }
+
+ $taken = { sid => $sid, source => $source, imbalance => $actual };
+ last LIMIT;
+ }
+ }
+ last if !$taken;
+
+ my $sid = $taken->{sid};
+ $origin->{$sid} //= $taken->{source};
+ $first_step->{$sid} //= $step;
+ $largest_gain = max($largest_gain, $current - $taken->{imbalance});
+ $current = $taken->{imbalance};
+ }
+
+ my $guest_stats = $work->{guest_stats};
+
+ if ($before < $balanced_imbalance && $before - $current < $min_total_ratio * $before) {
+ return {
+ mode => $mode,
+ imbalance => { before => $before, after => $before },
+ moves => [],
+ };
+ }
+
+ my @moved = sort { $first_step->{$a} <=> $first_step->{$b} } keys %$first_step;
+ my @moves =
+ map { { sid => $_, source => $origin->{$_}, target => $guest_stats->{$_}->{node} } }
+ grep { $guest_stats->{$_}->{node} ne $origin->{$_} } @moved;
+
+ return {
+ mode => $mode,
+ imbalance => { before => $before, after => $current },
+ moves => \@moves,
+ };
+}
+
+1;
diff --git a/src/PVE/ResourceScheduling/Scheduler.pm b/src/PVE/ResourceScheduling/Scheduler.pm
new file mode 100644
index 0000000..00ccc03
--- /dev/null
+++ b/src/PVE/ResourceScheduling/Scheduler.pm
@@ -0,0 +1,164 @@
+package PVE::ResourceScheduling::Scheduler;
+
+use v5.36;
+
+use PVE::RS::ResourceScheduling::Static;
+use PVE::RS::ResourceScheduling::Dynamic;
+
+# A thin mode-aware wrapper over PVE::RS::ResourceScheduling::Static / ::Dynamic that exposes the
+# incremental interface the HA stack drives directly (add a node, add/remove a resource one at a
+# time, then query). It hides that the two Rust backends name some methods differently and take
+# different arguments, so callers drive either mode through the same calls.
+#
+# This is deliberately a pure forwarder: it does NOT reconcile dynamic node load against the running
+# guests' usage; PVE::ResourceScheduling::Planner does that on its snapshot. Every call makes the
+# same scheduling decisions as calling the Rust backend directly.
+
+=head3 PVE::ResourceScheduling::Scheduler->new($mode)
+
+Construct a scheduler for C<$mode>, either 'static' or 'dynamic', building the matching
+C<PVE::RS::ResourceScheduling> backend. Dies on an unknown mode.
+
+=cut
+
+sub new($class, $mode) {
+ die "invalid scheduling mode '$mode'\n" if $mode ne 'static' && $mode ne 'dynamic';
+
+ my $backend =
+ $mode eq 'static'
+ ? PVE::RS::ResourceScheduling::Static->new()
+ : PVE::RS::ResourceScheduling::Dynamic->new();
+
+ return bless { mode => $mode, backend => $backend }, $class;
+}
+
+=head3 $scheduler->mode()
+
+The scheduling mode in use, 'static' or 'dynamic'.
+
+=cut
+
+sub mode($self) {
+ return $self->{mode};
+}
+
+=head3 $scheduler->add_node($node, $stats)
+
+Register C<$node> with the C<< { maxcpu, maxmem, cpu, mem } >> stats hash. Static mode uses only
+C<maxcpu>/C<maxmem> (coerced to integers); dynamic mode passes the full hash through. The shape is
+the same in both modes, so callers stay mode-agnostic.
+
+=cut
+
+sub add_node($self, $node, $stats) {
+ if ($self->{mode} eq 'static') {
+ $self->{backend}->add_node($node, int($stats->{maxcpu}), int($stats->{maxmem}));
+ } else {
+ $self->{backend}->add_node($node, $stats);
+ }
+
+ return;
+}
+
+=head3 $scheduler->remove_node($node)
+
+Remove C<$node> from the scheduler.
+
+=cut
+
+sub remove_node($self, $node) {
+ return $self->{backend}->remove_node($node);
+}
+
+=head3 $scheduler->list_nodes()
+
+The registered node names as a list.
+
+=cut
+
+sub list_nodes($self) {
+ return $self->{backend}->list_nodes()->@*;
+}
+
+=head3 $scheduler->contains_node($node)
+
+Whether C<$node> is registered.
+
+=cut
+
+sub contains_node($self, $node) {
+ return $self->{backend}->contains_node($node);
+}
+
+=head3 $scheduler->add_resource($sid, $resource)
+
+Add resource C<$sid> with the C<< { stats, running, 'current-node', 'target-node' } >> description
+C<$resource>.
+
+=cut
+
+sub add_resource($self, $sid, $resource) {
+ if ($self->{mode} eq 'static') {
+ $self->{backend}->add_service($sid, $resource);
+ } else {
+ $self->{backend}->add_resource($sid, $resource);
+ }
+
+ return;
+}
+
+=head3 $scheduler->remove_resource($sid)
+
+Remove resource C<$sid>'s usage from the nodes it was placed on.
+
+=cut
+
+sub remove_resource($self, $sid) {
+ if ($self->{mode} eq 'static') {
+ $self->{backend}->remove_service_usage($sid);
+ } else {
+ $self->{backend}->remove_resource($sid);
+ }
+
+ return;
+}
+
+=head3 $scheduler->calculate_node_imbalance()
+
+The current cluster load imbalance, as reported by the backend.
+
+=cut
+
+sub calculate_node_imbalance($self) {
+ return $self->{backend}->calculate_node_imbalance();
+}
+
+=head3 $scheduler->score_nodes_to_start($usage)
+
+Score every node for placing a resource of C<$usage>. Returns the raw backend result: an arrayref of
+C<< [$node, $score] >> pairs (higher score is better).
+
+=cut
+
+sub score_nodes_to_start($self, $usage) {
+ return $self->{mode} eq 'static'
+ ? $self->{backend}->score_nodes_to_start_service($usage)
+ : $self->{backend}->score_nodes_to_start_resource($usage);
+}
+
+=head3 $scheduler->score_best_balancing_migration_candidates($candidates, $limit, $method)
+
+Rank the migration C<$candidates> (each C<< { leader, resources, nodes } >>) and return up to
+C<$limit> best. The optional C<$method> selects the scorer: 'bruteforce' (default) or 'topsis'.
+
+=cut
+
+sub score_best_balancing_migration_candidates($self, $candidates, $limit, $method = 'bruteforce') {
+ die "invalid scoring method '$method'\n" if $method ne 'bruteforce' && $method ne 'topsis';
+
+ return $method eq 'topsis'
+ ? $self->{backend}->score_best_balancing_migration_candidates_topsis($candidates, $limit)
+ : $self->{backend}->score_best_balancing_migration_candidates($candidates, $limit);
+}
+
+1;
diff --git a/src/PVE/ResourceScheduling/Stats.pm b/src/PVE/ResourceScheduling/Stats.pm
new file mode 100644
index 0000000..1725407
--- /dev/null
+++ b/src/PVE/ResourceScheduling/Stats.pm
@@ -0,0 +1,194 @@
+package PVE::ResourceScheduling::Stats;
+
+use v5.36;
+
+use JSON qw(decode_json);
+
+use PVE::Cluster;
+
+# RRD entry indices for VM and CT guests, taken from
+# PVE::Service::pvestatd::update_{lxc,qemu}_status()
+use constant {
+ RRD_VM_INDEX_STATUS => 2,
+ RRD_VM_INDEX_MAXCPU => 5,
+ RRD_VM_INDEX_CPU => 6,
+ RRD_VM_INDEX_MAXMEM => 7,
+ RRD_VM_INDEX_MEM => 8,
+ RRD_VM_INDEX_MEMHOST => 15,
+};
+
+# RRD entry indices for PVE nodes, taken from
+# PVE::Service::pvestatd::update_node_status()
+use constant {
+ RRD_NODE_INDEX_MAXCPU => 4,
+ RRD_NODE_INDEX_CPU => 5,
+ RRD_NODE_INDEX_MAXMEM => 7,
+ RRD_NODE_INDEX_MEM => 8,
+};
+
+my $HOSTNAME_RE = qr/(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]{,61}?[a-zA-Z0-9])?)/;
+
+my $guest_type_map = {
+ qemu => 'vm',
+ lxc => 'ct',
+};
+
+# Enumerate the cluster guests with their run state taken from $rrd, a PVE::Cluster::rrd_dump()
+# result. Callers that also read usage from the dump pass in the same one, so run state and usage
+# always come from one consistent snapshot.
+my sub cluster_guest_stats($rrd) {
+ my $vmlist = PVE::Cluster::get_vmlist();
+ my $idlist = $vmlist->{ids} // {};
+
+ my $stats = {};
+ for my $id (keys %$idlist) {
+ my $type = $guest_type_map->{ $idlist->{$id}->{type} // '' };
+ next if !defined($type); # silently ignore unknown guest types
+
+ my $sid = "$type:$id";
+
+ my $rrdentry = $rrd->{"pve-vm-9.0/$id"} // [];
+ # can be any QMP RunState, but 'running' is the only active guest state
+ my $status = $rrdentry->[RRD_VM_INDEX_STATUS] // 'stopped';
+ my $state = $status eq 'running' ? 'started' : 'stopped';
+
+ $stats->{$sid} = {
+ id => $id,
+ node => $idlist->{$id}->{node},
+ type => $type,
+ state => $state,
+ running => $state eq 'started' ? 1 : 0,
+ usage => {},
+ };
+ }
+
+ return $stats;
+}
+
+=head3 get_cluster_guest_stats()
+
+Return all cluster guests as C<< { "$type:$id" => { id, node, type, state, running, usage } } >>
+with empty usage. Useful on its own for callers that only need the guest list and run state; the
+gatherers below fill in the usage.
+
+=cut
+
+sub get_cluster_guest_stats() {
+ return cluster_guest_stats(PVE::Cluster::rrd_dump());
+}
+
+=head3 get_dynamic_guest_stats()
+
+Cluster guest stats with each guest's measured (live, RRD-based) usage filled in, as
+C<< { cpu, maxcpu, mem, maxmem } >>. For VMs, C<mem> is the host-side memory usage.
+
+=cut
+
+sub get_dynamic_guest_stats() {
+ my $rrd = PVE::Cluster::rrd_dump();
+
+ my $stats = cluster_guest_stats($rrd);
+ for my $sid (keys %$stats) {
+ my $id = $stats->{$sid}->{id};
+ my $rrdentry = $rrd->{"pve-vm-9.0/$id"} // [];
+
+ # NOTE the guests' broadcasted vmstatus() caps maxcpu at the node's maxcpu
+ my $maxcpu = ($rrdentry->[RRD_VM_INDEX_MAXCPU] || 0.0) + 0.0;
+
+ # host memory usage is the real cost of a VM on its node and matches what the node stats
+ # account for; containers do not report it, so fall back to their memory usage
+ my $mem = $rrdentry->[RRD_VM_INDEX_MEMHOST] // $rrdentry->[RRD_VM_INDEX_MEM];
+
+ $stats->{$sid}->{usage} = {
+ maxcpu => $maxcpu,
+ cpu => (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
+ maxmem => int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0),
+ mem => int($mem || 0),
+ };
+ }
+
+ return $stats;
+}
+
+=head3 get_static_guest_stats($resolvers)
+
+Cluster guest stats with each guest's configured C<< { maxcpu, maxmem } >> usage filled in.
+C<$resolvers> maps a guest type ('vm' or 'ct') to a code ref returning that usage for a guest config
+hash; the caller passes its own so this module stays free of guest-type dependencies. Dies if a
+guest's type has no resolver.
+
+=cut
+
+sub get_static_guest_stats($resolvers) {
+ my $properties = ['cores', 'cpulimit', 'memory', 'sockets', 'vcpus'];
+
+ my $stats = get_cluster_guest_stats();
+ my $confs = PVE::Cluster::get_guest_config_properties($properties);
+
+ for my $sid (keys %$stats) {
+ my ($id, $type) = $stats->{$sid}->@{qw(id type)};
+
+ my $resolver = $resolvers->{$type}
+ or die "no static stats resolver for guest type '$type'\n";
+
+ $stats->{$sid}->{usage} = $resolver->($confs->{$id} // {});
+ }
+
+ return $stats;
+}
+
+=head3 get_dynamic_node_stats()
+
+Return measured (live, RRD-based) per-node usage as
+C<< { $node => { cpu, maxcpu, mem, maxmem } } >>.
+
+=cut
+
+sub get_dynamic_node_stats() {
+ my $rrd = PVE::Cluster::rrd_dump();
+
+ my $stats = {};
+ for my $key (keys %$rrd) {
+ my ($nodename) = $key =~ m/^pve-node-9\.0\/($HOSTNAME_RE)$/;
+ next if !$nodename;
+
+ my $rrdentry = $rrd->{$key} // [];
+
+ my $maxcpu = int($rrdentry->[RRD_NODE_INDEX_MAXCPU] || 0);
+
+ $stats->{$nodename} = {
+ maxcpu => $maxcpu,
+ cpu => (($rrdentry->[RRD_NODE_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
+ maxmem => int($rrdentry->[RRD_NODE_INDEX_MAXMEM] || 0),
+ mem => int($rrdentry->[RRD_NODE_INDEX_MEM] || 0),
+ };
+ }
+
+ return $stats;
+}
+
+=head3 get_static_node_stats()
+
+Return configured per-node capacity as C<< { $node => { maxcpu, maxmem } } >>, decoded from the
+'static-info' node key-value broadcast.
+
+=cut
+
+sub get_static_node_stats() {
+ my $stats = PVE::Cluster::get_node_kv('static-info');
+
+ for my $node (keys %$stats) {
+ $stats->{$node} = eval {
+ my $node_stats = decode_json($stats->{$node});
+ return {
+ maxcpu => $node_stats->{cpus},
+ maxmem => $node_stats->{memory},
+ };
+ };
+ warn "unable to decode static node info for '$node' - $@" if $@;
+ }
+
+ return $stats;
+}
+
+1;
diff --git a/src/tests/Makefile b/src/tests/Makefile
index 8eb38a8..5d1bde9 100644
--- a/src/tests/Makefile
+++ b/src/tests/Makefile
@@ -1,8 +1,16 @@
all:
-.PHONY: check abstract-config guest-id hookscript
-check: abstract-config guest-id hookscript
+TESTS = \
+ abstract-config \
+ guest-id \
+ hookscript \
+ resource-scheduling-planner \
+ resource-scheduling-scheduler \
+ resource-scheduling-stats
+
+.PHONY: check $(TESTS)
+check: $(TESTS)
abstract-config: abstract-config-tests.pl
perl -I.. ./abstract-config-tests.pl
@@ -12,3 +20,12 @@ guest-id: guest-id-tests.pl
hookscript: hookscript-tests.pl
perl -I.. ./hookscript-tests.pl
+
+resource-scheduling-planner: resource-scheduling-planner-tests.pl
+ perl -I.. ./resource-scheduling-planner-tests.pl
+
+resource-scheduling-scheduler: resource-scheduling-scheduler-tests.pl
+ perl -I.. ./resource-scheduling-scheduler-tests.pl
+
+resource-scheduling-stats: resource-scheduling-stats-tests.pl
+ perl -I.. ./resource-scheduling-stats-tests.pl
diff --git a/src/tests/resource-scheduling-planner-tests.pl b/src/tests/resource-scheduling-planner-tests.pl
new file mode 100644
index 0000000..60ed048
--- /dev/null
+++ b/src/tests/resource-scheduling-planner-tests.pl
@@ -0,0 +1,1775 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+use List::Util qw(max min);
+use Test::More;
+
+use PVE::ResourceScheduling::Planner;
+use PVE::ResourceScheduling::Scheduler;
+
+my $GB = 1024 * 1024 * 1024;
+my $EPS = 1e-9;
+
+# helper: a homogeneous set of nodes with no base load
+my sub nodes {
+ my ($count, %stats) = @_;
+ my $node_stats = {};
+ for my $i (1 .. $count) {
+ $node_stats->{"node$i"} = {
+ maxcpu => 8,
+ cpu => 0,
+ maxmem => 16 * $GB,
+ mem => 0,
+ %stats,
+ };
+ }
+ return $node_stats;
+}
+
+my sub guest {
+ my ($node, %usage) = @_;
+ return {
+ node => $node,
+ running => 1,
+ usage => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0, %usage },
+ };
+}
+
+# build a snapshot Planner over exactly the nodes named by the node-stats keys
+my sub planner {
+ my ($mode, $node_stats, $guest_stats) = @_;
+ return PVE::ResourceScheduling::Planner->new(
+ mode => $mode,
+ nodes => [sort keys %$node_stats],
+ node_stats => $node_stats,
+ guest_stats => $guest_stats // {},
+ );
+}
+
+# Apply a proposal's moves to a copy of the snapshot and return a fresh Planner over the result. In
+# dynamic mode a guest's measured usage moves with it.
+my sub replay {
+ my ($mode, $node_stats, $guest_stats, $moves) = @_;
+
+ my $ns = { map { $_ => { $node_stats->{$_}->%* } } keys %$node_stats };
+ my $gs = { map { $_ => { $guest_stats->{$_}->%* } } keys %$guest_stats };
+
+ for my $move (@$moves) {
+ my ($sid, $source, $target) = $move->@{qw(sid source target)};
+ is($gs->{$sid}->{node}, $source, "replayed move of $sid starts on its source");
+
+ if ($mode eq 'dynamic') {
+ my $usage = $gs->{$sid}->{usage};
+ $ns->{$source}->{cpu} = max(0, $ns->{$source}->{cpu} - $usage->{cpu});
+ $ns->{$source}->{mem} = max(0, $ns->{$source}->{mem} - $usage->{mem});
+ $ns->{$target}->{cpu} += $usage->{cpu};
+ $ns->{$target}->{mem} += $usage->{mem};
+ }
+ $gs->{$sid}->{node} = $target;
+ }
+
+ return planner($mode, $ns, $gs);
+}
+
+# Compare two rebalance proposals: the moves must match exactly, the imbalance values up to the
+# rounding noise of the backend, which sums up the node loads in no fixed order.
+my sub plan_is {
+ my ($got, $expected, $name) = @_;
+ my $close = sub {
+ my ($key) = @_;
+ return abs($got->{imbalance}->{$key} - $expected->{imbalance}->{$key}) < $EPS ? 1 : 0;
+ };
+ is_deeply(
+ [$got->{mode}, $got->{moves}, $close->('before'), $close->('after')],
+ [$expected->{mode}, $expected->{moves}, 1, 1],
+ $name,
+ );
+}
+
+# The pairs of guests from $separate (in either direction) that share a node once the proposal's
+# moves are applied, as "sid/sid" strings.
+my sub shared_nodes {
+ my ($guest_stats, $moves, $separate) = @_;
+ my $node = { map { $_ => $guest_stats->{$_}->{node} } keys %$guest_stats };
+ $node->{ $_->{sid} } = $_->{target} for @$moves;
+
+ my @together;
+ for my $sid (sort keys %$separate) {
+ for my $partner ($separate->{$sid}->@*) {
+ push @together, "$sid/$partner" if $node->{$sid} eq $node->{$partner};
+ }
+ }
+ return \@together;
+}
+
+# A deterministic cluster with an overloaded first node, some unattributed node overhead and some
+# nodes measuring less than their guests report, so the dynamic reconcile matters.
+my sub synthetic_cluster {
+ my ($node_count, $guest_count) = @_;
+
+ my @nodes = map { "node$_" } 1 .. $node_count;
+ my $node_stats = {
+ map {
+ my $i = $_;
+ (
+ $nodes[$i] => {
+ maxcpu => (16, 32, 64)[$i % 3],
+ maxmem => (64, 128, 256)[($i + 1) % 3] * $GB,
+ cpu => 0,
+ mem => 0,
+ },
+ );
+ } 0 .. $#nodes
+ };
+
+ my $guest_stats = {};
+ for my $i (1 .. $guest_count) {
+ my $node = $i % 3 == 0 ? $nodes[$i % $node_count] : $nodes[0];
+ my $running = $i % 7 != 0;
+ my $usage = {
+ maxcpu => 0.0 + (1, 2, 4)[$i % 3],
+ cpu => $running ? (($i * 7) % 10) / 4 : 0,
+ maxmem => (2, 4, 8, 16)[$i % 4] * $GB,
+ mem => $running ? (($i * 5) % 8 + 1) * $GB / 2 : 0,
+ };
+ $guest_stats->{ "vm:" . (100 + $i) } =
+ { node => $node, running => $running ? 1 : 0, usage => $usage };
+
+ if ($running) {
+ $node_stats->{$node}->{cpu} += $usage->{cpu};
+ $node_stats->{$node}->{mem} += $usage->{mem};
+ }
+ }
+
+ for my $i (0 .. $#nodes) {
+ my $stats = $node_stats->{ $nodes[$i] };
+ if ($i % 3 == 1) { # measures less than its guests report
+ $stats->{cpu} *= 0.6;
+ $stats->{mem} = int($stats->{mem} * 0.7);
+ } else { # host overhead on top of the guests
+ $stats->{cpu} += 0.5 * $i;
+ $stats->{mem} += $i * $GB;
+ }
+ }
+
+ return ($node_stats, $guest_stats);
+}
+
+subtest 'node_imbalance reflects the load distribution' => sub {
+ my $stacked = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 6, maxmem => 16 * $GB, mem => 12 * $GB },
+ node2 => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ node3 => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ },
+ {
+ 'vm:100' => guest('node1', cpu => 3, mem => 6 * $GB),
+ 'vm:101' => guest('node1', cpu => 3, mem => 6 * $GB),
+ },
+ );
+
+ my $balanced = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ node2 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ node3 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ },
+ );
+
+ cmp_ok($stacked->node_imbalance(), '>', 0.5, 'stacked load yields high imbalance');
+ cmp_ok($balanced->node_imbalance(), '<', 0.01, 'even load yields near-zero imbalance');
+};
+
+subtest 'reconcile: host overhead (ARC) stays counted as load' => sub {
+ my $without_base = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ node2 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ },
+ {
+ 'vm:100' => guest('node1', cpu => 2, mem => 4 * $GB),
+ 'vm:101' => guest('node2', cpu => 2, mem => 4 * $GB),
+ },
+ );
+
+ # node1 additionally carries 6 GiB of non-guest memory (e.g. ZFS ARC)
+ my $with_base = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 10 * $GB },
+ node2 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ },
+ {
+ 'vm:100' => guest('node1', cpu => 2, mem => 4 * $GB),
+ 'vm:101' => guest('node2', cpu => 2, mem => 4 * $GB),
+ },
+ );
+
+ cmp_ok(
+ $with_base->node_imbalance(),
+ '>',
+ $without_base->node_imbalance(),
+ 'unattributed host overhead is kept as real node load',
+ );
+};
+
+subtest 'reconcile: node load never drops below the sum of its running guests' => sub {
+ # all nodes carry load, so the imbalance reacts to every node's value
+ my $peers = {
+ node2 => { maxcpu => 8, cpu => 3, maxmem => 16 * $GB, mem => 6 * $GB },
+ node3 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 5 * $GB },
+ };
+ my $guests = { 'vm:100' => guest('node1', cpu => 2, mem => 4 * $GB) };
+
+ # node1 measures less cpu and memory than its running guest reports
+ my $under = { maxcpu => 8, cpu => 1, maxmem => 16 * $GB, mem => 2 * $GB };
+ my $under_reported = planner('dynamic', { %$peers, node1 => $under }, $guests);
+
+ my $consistent = planner(
+ 'dynamic',
+ { %$peers, node1 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB } },
+ $guests,
+ );
+
+ # the same measured load without any guest: what the raw values alone amount to
+ my $raw = planner('dynamic', { %$peers, node1 => $under });
+
+ cmp_ok(
+ abs($raw->node_imbalance() - $consistent->node_imbalance()),
+ '>',
+ 0.01,
+ 'the raw node load is distinguishable from the guest sum',
+ );
+ ok(
+ abs($under_reported->node_imbalance() - $consistent->node_imbalance()) < $EPS,
+ 'an under-reported node is lifted to the sum of its running guests',
+ );
+
+ # a stopped guest puts no load on its node, whatever usage it last reported
+ my $with_stopped = planner(
+ 'dynamic',
+ { %$peers, node1 => $under },
+ {
+ %$guests,
+ 'vm:101' => { guest('node1', cpu => 4, mem => 8 * $GB)->%*, running => 0 },
+ },
+ );
+ ok(
+ abs($with_stopped->node_imbalance() - $consistent->node_imbalance()) < $EPS,
+ 'stopped guests do not count toward the guest sum',
+ );
+};
+
+subtest 'rank_target_nodes: running guest is ranked as a migration' => sub {
+ my $node_stats = {
+ node1 => { maxcpu => 8, cpu => 4, maxmem => 16 * $GB, mem => 8 * $GB },
+ node2 => { maxcpu => 8, cpu => 3, maxmem => 16 * $GB, mem => 6 * $GB },
+ node3 => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ };
+ my $guest_stats = {
+ 'vm:100' => guest('node1', cpu => 4, mem => 8 * $GB),
+ 'vm:200' => guest('node2', cpu => 3, mem => 6 * $GB),
+ };
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+
+ my $ranked = $planner->rank_target_nodes('vm:100');
+
+ is_deeply([map { $_->{node} } @$ranked], ['node3', 'node2'], 'emptier node ranks first');
+ for my $entry (@$ranked) {
+ my $moved = [{ sid => 'vm:100', source => 'node1', target => $entry->{node} }];
+ my $expected = replay('dynamic', $node_stats, $guest_stats, $moved)->node_imbalance();
+ ok(
+ abs($entry->{imbalance} - $expected) < $EPS,
+ "imbalance for $entry->{node} is the imbalance after migrating there",
+ ) or diag("ranked $entry->{imbalance}, replayed $expected");
+ }
+};
+
+subtest 'rank_target_nodes: stopped guest is ranked as a placement' => sub {
+ my $planner = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 1, maxmem => 16 * $GB, mem => 2 * $GB },
+ node2 => { maxcpu => 8, cpu => 6, maxmem => 16 * $GB, mem => 12 * $GB },
+ node3 => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ },
+ {
+ 'vm:100' => guest('node2', cpu => 6, mem => 12 * $GB),
+ 'vm:200' => {
+ node => 'node1',
+ running => 0,
+ usage => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ },
+ },
+ );
+
+ my $ranked = $planner->rank_target_nodes('vm:200');
+
+ is_deeply(
+ [map { $_->{node} } @$ranked],
+ ['node3', 'node2'],
+ 'placement ranking excludes the current node and prefers the emptier node',
+ );
+ ok(!(grep { defined($_->{imbalance}) } @$ranked), 'placement entries carry no imbalance');
+};
+
+subtest 'rank_target_nodes: candidate restriction' => sub {
+ my $planner = PVE::ResourceScheduling::Planner->new(
+ mode => 'dynamic',
+ nodes => [qw(node1 node2 node3 node4 node5)],
+ node_stats => {
+ nodes(4, cpu => 1, mem => 2 * $GB)->%*,
+ node5 => { maxcpu => 0, maxmem => 16 * $GB }, # unusable, skipped
+ },
+ guest_stats => {
+ 'vm:100' => guest('node1', cpu => 2, mem => 4 * $GB),
+ 'vm:101' => { guest('node2')->%*, running => 0 },
+ },
+ );
+
+ my $restriction = [qw(node3 node3 node2 node1 node5 nosuchnode)];
+
+ is_deeply(
+ [map { $_->{node} } $planner->rank_target_nodes('vm:100', $restriction)->@*],
+ ['node2', 'node3'],
+ 'migration ranking keeps only the listed snapshot nodes other than the current one, once',
+ );
+ is_deeply(
+ [sort map { $_->{node} } $planner->rank_target_nodes('vm:101', $restriction)->@*],
+ ['node1', 'node3'],
+ 'placement ranking keeps only the listed snapshot nodes other than the current one, once',
+ );
+ is_deeply($planner->rank_target_nodes('vm:100', []), [], 'empty restriction ranks nothing');
+ is_deeply(
+ $planner->rank_target_nodes('vm:101', ['node2']),
+ [],
+ 'restriction to the current node ranks nothing',
+ );
+};
+
+subtest 'rank_target_nodes: a running guest on an offline node falls back to placement' => sub {
+ # the guest's node is offline (absent from the snapshot), so it must rank as a placement
+ my $planner = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 6, maxmem => 16 * $GB, mem => 12 * $GB },
+ node2 => { maxcpu => 8, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ },
+ {
+ 'vm:100' => guest('offline-node', cpu => 2, mem => 4 * $GB),
+ },
+ );
+
+ my $ranked = eval { $planner->rank_target_nodes('vm:100') };
+ ok(!$@, 'ranking a guest on an offline node does not die') or diag $@;
+ is_deeply(
+ [map { $_->{node} } @$ranked],
+ ['node2', 'node1'],
+ 'placement ranking covers all snapshot nodes, emptier first',
+ );
+ ok(!(grep { defined($_->{imbalance}) } @$ranked), 'placement entries carry no imbalance');
+};
+
+subtest 'rank_target_nodes: dies for an unknown sid' => sub {
+ my $planner = planner('dynamic', nodes(1));
+ eval { $planner->rank_target_nodes('vm:999') };
+ like($@, qr/no such guest 'vm:999'/, 'an unknown sid is rejected');
+};
+
+subtest 'nodes without usable stats are skipped' => sub {
+ my $planner = PVE::ResourceScheduling::Planner->new(
+ mode => 'dynamic',
+ nodes => [qw(node1 node2 node3 node4 node5 node6)],
+ node_stats => {
+ node1 => { maxcpu => 8, cpu => 6, maxmem => 16 * $GB, mem => 12 * $GB },
+ node2 => { maxcpu => 8, maxmem => 16 * $GB }, # no measured load: idle
+ # node3 has no stats at all
+ node4 => { maxcpu => 0, cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ node5 => { maxcpu => 8, cpu => 0, maxmem => 0, mem => 0 },
+ node6 => { cpu => 0, maxmem => 16 * $GB, mem => 0 },
+ offline => undef, # not requested, e.g. undecodable info of an offline node
+ },
+ guest_stats => {
+ 'vm:100' => guest('node1', cpu => 3, mem => 6 * $GB),
+ 'vm:101' => guest('node1', cpu => 3, mem => 6 * $GB),
+ 'vm:300' => guest('node3', cpu => 2, mem => 4 * $GB),
+ },
+ );
+
+ is_deeply(
+ [map { $_->{node} } $planner->rank_target_nodes('vm:100')->@*],
+ ['node2'],
+ 'skipped nodes are no migration candidates',
+ );
+
+ my $ranked = $planner->rank_target_nodes('vm:300');
+ ok(
+ !(grep { defined($_->{imbalance}) } @$ranked),
+ 'a guest on a skipped node is ranked as a placement',
+ );
+ is_deeply(
+ [sort map { $_->{node} } @$ranked],
+ [qw(node1 node2)],
+ 'placement covers the usable nodes only',
+ );
+
+ my $plan = eval { $planner->suggest_rebalance([qw(vm:100 vm:101 vm:300)]) };
+ ok(!$@, 'rebalancing with an undef stats entry for an unrequested node works') or diag $@;
+ is_deeply(
+ $plan->{moves},
+ [{ sid => 'vm:100', source => 'node1', target => 'node2' }],
+ 'rebalance only moves between usable nodes and leaves the skipped node\'s guest alone',
+ );
+};
+
+subtest 'the snapshot is copied on construction' => sub {
+ my $node_stats = nodes(3);
+ $node_stats->{node1}->@{qw(cpu mem)} = (6, 12 * $GB);
+ my $guest_stats = {};
+ $guest_stats->{"vm:10$_"} = guest('node1', cpu => 2, mem => 4 * $GB) for 0 .. 2;
+
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $imbalance = $planner->node_imbalance();
+ my $plan = $planner->suggest_rebalance([sort keys %$guest_stats]);
+ is(scalar($plan->{moves}->@*), 2, 'the stacked guests are spread out');
+
+ # changes the planner must not see, neither in its initial scheduler nor in rebuilt ones
+ $node_stats->{node2}->{mem} = 15 * $GB;
+ for my $guest (values %$guest_stats) {
+ $guest->{node} = 'node3';
+ $guest->{usage}->@{qw(cpu mem)} = (0, 0);
+ }
+
+ is($planner->node_imbalance(), $imbalance, 'imbalance unaffected by caller changes');
+ plan_is(
+ $planner->suggest_rebalance([sort keys %$guest_stats]),
+ $plan,
+ 'proposal unaffected by caller changes',
+ );
+};
+
+subtest 'suggest_rebalance: greedily descends toward balance, state-free' => sub {
+ my $guest_stats = {};
+ $guest_stats->{"vm:10$_"} = guest('node1', cpu => 1, mem => 2 * $GB) for (0 .. 5);
+
+ my $node_stats = nodes(3);
+ $node_stats->{node1}->{cpu} = 6;
+ $node_stats->{node1}->{mem} = 12 * $GB;
+
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $imbalance = $planner->node_imbalance();
+
+ my $plan = $planner->suggest_rebalance([sort keys %$guest_stats]);
+
+ is($plan->{mode}, 'dynamic', 'the mode is reported');
+ is(scalar($plan->{moves}->@*), 4, 'two guests move to each of the idle nodes');
+ ok(abs($plan->{imbalance}->{before} - $imbalance) < $EPS,
+ 'before is the current imbalance');
+ ok($plan->{imbalance}->{after} < $EPS, 'the proposal balances the cluster');
+ ok(
+ !(grep { $_->{source} ne 'node1' || $_->{target} eq 'node1' } $plan->{moves}->@*),
+ 'all moves go from the stacked node to another node',
+ );
+
+ # proposing works on a copy: the planner's own state is untouched, so asking again yields the
+ # same proposal
+ is($planner->node_imbalance(), $imbalance, 'planner imbalance unchanged by the proposal');
+ plan_is(
+ $planner->suggest_rebalance([sort keys %$guest_stats]),
+ $plan,
+ 'a second proposal is identical',
+ );
+
+ my $strict = $planner->suggest_rebalance([sort keys %$guest_stats], min_improvement => 0.9);
+ is_deeply($strict->{moves}, [], 'no move taken when none clears a high min_improvement');
+ is(
+ $strict->{imbalance}->{after},
+ $strict->{imbalance}->{before},
+ 'after equals before without moves',
+ );
+
+ my $restricted = $planner->suggest_rebalance(['vm:100', 'vm:999']);
+ is_deeply(
+ [map { $_->{sid} } $restricted->{moves}->@*],
+ ['vm:100'],
+ 'only the movable guests are moved, unknown sids are ignored',
+ );
+};
+
+subtest 'suggest_rebalance: proposals replay to the reported imbalance' => sub {
+ for my $mode (qw(dynamic static)) {
+ for my $shape ([3, 30], [5, 60]) {
+ my ($node_stats, $guest_stats) = synthetic_cluster(@$shape);
+ if ($mode eq 'static') {
+ $node_stats =
+ { map { $_ => { $node_stats->{$_}->%{qw(maxcpu maxmem)} } } keys %$node_stats };
+ $guest_stats->{$_}->{usage} =
+ { $guest_stats->{$_}->{usage}->%{qw(maxcpu maxmem)} }
+ for keys %$guest_stats;
+ }
+ my $planner = planner($mode, $node_stats, $guest_stats);
+
+ for my $method (qw(bruteforce topsis)) {
+ my $plan = $planner->suggest_rebalance(
+ [sort keys %$guest_stats],
+ max_moves => 50,
+ method => $method,
+ );
+ my $name = "$mode/$method, $shape->[0] nodes, $shape->[1] guests";
+
+ my $moves = $plan->{moves};
+ my %sids = map { $_->{sid} => 1 } @$moves;
+ is(scalar(keys %sids), scalar(@$moves), "$name: one move per guest");
+ ok(
+ !(grep { $_->{source} eq $_->{target} } @$moves),
+ "$name: no move ends on its source",
+ );
+ ok(
+ !(grep { !$guest_stats->{ $_->{sid} }->{running} } @$moves),
+ "$name: only running guests are moved",
+ );
+
+ my $replayed =
+ replay($mode, $node_stats, $guest_stats, $moves)->node_imbalance();
+ ok(
+ abs($replayed - $plan->{imbalance}->{after}) < $EPS,
+ "$name: after matches the replayed proposal",
+ ) or diag("after $plan->{imbalance}->{after}, replayed $replayed");
+ cmp_ok(
+ $plan->{imbalance}->{after},
+ '<',
+ $plan->{imbalance}->{before},
+ "$name: the proposal improves the imbalance",
+ );
+ }
+ }
+ }
+};
+
+subtest 'suggest_rebalance: a guest moved repeatedly is reported once' => sub {
+ # without overhead, the greedy steps are vm:103 node1->node2, vm:104 node1->node3, vm:101
+ # node4->node3 and vm:104 node3->node2
+ my $usage =
+ sub { return (maxcpu => 4, maxmem => 32 * $GB, cpu => $_[0], mem => $_[1] * $GB) };
+ my $guest_stats = {
+ 'vm:101' => guest('node4', $usage->(7, 7)),
+ 'vm:102' => guest('node4', $usage->(6, 13)),
+ 'vm:103' => guest('node1', $usage->(6, 6)),
+ 'vm:104' => guest('node1', $usage->(6, 4)),
+ 'vm:105' => guest('node1', $usage->(2, 15)),
+ 'vm:106' => guest('node1', $usage->(3, 12)),
+ };
+ my $node_stats = nodes(4, maxcpu => 16, maxmem => 64 * $GB);
+ for my $guest (values %$guest_stats) {
+ $node_stats->{ $guest->{node} }->{cpu} += $guest->{usage}->{cpu};
+ $node_stats->{ $guest->{node} }->{mem} += $guest->{usage}->{mem};
+ }
+
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $plan = $planner->suggest_rebalance([sort keys %$guest_stats]);
+
+ is_deeply(
+ $plan->{moves},
+ [
+ { sid => 'vm:103', source => 'node1', target => 'node2' },
+ { sid => 'vm:104', source => 'node1', target => 'node2' },
+ { sid => 'vm:101', source => 'node4', target => 'node3' },
+ ],
+ 'the double hop collapses into one move from the original to the final node',
+ );
+
+ my $replayed =
+ replay('dynamic', $node_stats, $guest_stats, $plan->{moves})->node_imbalance();
+ ok(abs($replayed - $plan->{imbalance}->{after}) < $EPS,
+ 'after matches the collapsed moves');
+
+ # the step cap counts greedy steps, collapsed moves can be fewer
+ is(
+ scalar(
+ $planner->suggest_rebalance([sort keys %$guest_stats], max_moves => 4)->{moves}->@*
+ ),
+ 3,
+ 'four steps yield three moves',
+ );
+};
+
+subtest 'suggest_rebalance: no move in an already balanced cluster' => sub {
+ # moving any guest only permutes the node loads, which a truncated prediction could make look
+ # like a tiny improvement
+ my $node_stats = { map { ("node$_" => { maxcpu => 16, maxmem => 64 * $GB }) } 1 .. 3 };
+ my $guest_stats = {
+ map { (
+ "vm:"
+ . (100 + $_) => {
+ node => 'node' . (1 + $_ % 3),
+ running => 1,
+ usage => { maxcpu => 2, maxmem => 3 * $GB },
+ },
+ ) } 0 .. 6
+ };
+ my $plan =
+ planner('static', $node_stats, $guest_stats)->suggest_rebalance([sort keys %$guest_stats]);
+ is_deeply($plan->{moves}, [], 'no move for a load permutation');
+
+ # moving a guest without any usage changes nothing
+ my $idle = planner(
+ 'dynamic',
+ {
+ node1 => { maxcpu => 8, cpu => 1.7, maxmem => 16 * $GB, mem => 5 * $GB },
+ node2 => { maxcpu => 8, cpu => 4.3, maxmem => 16 * $GB, mem => 3 * $GB },
+ node3 => { maxcpu => 8, cpu => 2.9, maxmem => 16 * $GB, mem => 9 * $GB },
+ },
+ { 'vm:100' => guest('node1') },
+ );
+ is_deeply($idle->suggest_rebalance(['vm:100'])->{moves}, [], 'no move for an idle guest');
+};
+
+subtest 'suggest_rebalance: a mispredicted move is not taken' => sub {
+ # The static backend accounts a container without CPU limit with the full core count of its
+ # node, but migrates it as if it used no CPU. The second greedy step here is predicted to
+ # improve the imbalance, but actually worsens it. Its predicted gain is small compared to the
+ # first step, so disable the step ratio guard to see the actual-imbalance check at work.
+ my $node_stats = {
+ node1 => { maxcpu => 16, maxmem => 64 * $GB },
+ node2 => { maxcpu => 16, maxmem => 64 * $GB },
+ node3 => { maxcpu => 64, maxmem => 64 * $GB },
+ };
+ my $guest_stats = {
+ 'ct:100' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 0, maxmem => 4 * $GB } },
+ 'ct:101' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 0, maxmem => 4 * $GB } },
+ 'vm:102' =>
+ { node => 'node2', running => 1, usage => { maxcpu => 4, maxmem => 8 * $GB } },
+ };
+
+ my $plan = planner('static', $node_stats, $guest_stats)
+ ->suggest_rebalance([sort keys %$guest_stats], min_step_ratio => 0);
+
+ is_deeply(
+ $plan->{moves},
+ [{ sid => 'ct:100', source => 'node1', target => 'node3' }],
+ 'the proposal stops before the worsening step',
+ );
+ my $replayed =
+ replay('static', $node_stats, $guest_stats, $plan->{moves})->node_imbalance();
+ ok(abs($replayed - $plan->{imbalance}->{after}) < $EPS, 'after is the actual imbalance');
+
+ # The container is the best pick here, but moving it changes nothing, while a VM move pays off.
+ $node_stats = { map { ("node$_" => { maxcpu => 16, maxmem => 64 * $GB }) } 1 .. 5 };
+ $guest_stats = {
+ 'ct:100' =>
+ { node => 'node5', running => 1, usage => { maxcpu => 0, maxmem => 10 * $GB } },
+ 'vm:101' =>
+ { node => 'node3', running => 1, usage => { maxcpu => 1, maxmem => 11 * $GB } },
+ 'vm:102' =>
+ { node => 'node3', running => 1, usage => { maxcpu => 4, maxmem => 4 * $GB } },
+ };
+ $plan =
+ planner('static', $node_stats, $guest_stats)->suggest_rebalance([sort keys %$guest_stats]);
+ is_deeply(
+ $plan->{moves},
+ [{ sid => 'vm:101', source => 'node3', target => 'node1' }],
+ 'a step tries the next best guest after an undone move',
+ );
+};
+
+subtest 'suggest_rebalance: step limit and option validation' => sub {
+ my $guest_stats = {};
+ $guest_stats->{ "vm:" . (100 + $_) } = guest('node1', cpu => 1, mem => 1 * $GB) for 1 .. 30;
+ my $node_stats = nodes(3, maxcpu => 64, maxmem => 128 * $GB);
+ $node_stats->{node1}->@{qw(cpu mem)} = (30, 30 * $GB);
+
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ is(PVE::ResourceScheduling::Planner::DEFAULT_MAX_MOVES, 10, 'the default step limit is 10');
+ is(PVE::ResourceScheduling::Planner::DEFAULT_MIN_STEP_RATIO, 0.1, 'the default step ratio');
+ is(
+ PVE::ResourceScheduling::Planner::DEFAULT_MIN_TOTAL_RATIO,
+ 0.1,
+ 'the default total ratio',
+ );
+ is(
+ PVE::ResourceScheduling::Planner::DEFAULT_BALANCED_IMBALANCE,
+ 0.1,
+ 'the default balanced imbalance',
+ );
+ is(
+ PVE::ResourceScheduling::Planner::DEFAULT_BUSIEST_NODES,
+ 10,
+ 'the default candidate nodes',
+ );
+ is(
+ scalar($planner->suggest_rebalance($movable)->{moves}->@*),
+ PVE::ResourceScheduling::Planner::DEFAULT_MAX_MOVES,
+ 'the default step limit applies',
+ );
+
+ my $full = $planner->suggest_rebalance($movable, max_moves => 25);
+ is(scalar($full->{moves}->@*), 20, 'a higher limit balances the cluster completely');
+ ok($full->{imbalance}->{after} < $EPS, 'the full proposal reaches balance');
+
+ for my $max_moves (0, -1, 1.5, 'abc') {
+ eval { $planner->suggest_rebalance($movable, max_moves => $max_moves) };
+ like($@, qr/invalid max_moves/, "max_moves '$max_moves' is rejected");
+ }
+ for my $min_improvement (-0.1, 'abc', 'nan') {
+ eval { $planner->suggest_rebalance($movable, min_improvement => $min_improvement) };
+ like($@, qr/invalid min_improvement/, "min_improvement '$min_improvement' is rejected");
+ }
+ for my $ratio (-0.1, 1.5, 'abc', 'nan') {
+ eval { $planner->suggest_rebalance($movable, min_step_ratio => $ratio) };
+ like($@, qr/invalid min_step_ratio/, "min_step_ratio '$ratio' is rejected");
+ }
+ for my $total (-0.1, 1.5, 'abc', 'nan') {
+ eval { $planner->suggest_rebalance($movable, min_total_ratio => $total) };
+ like($@, qr/invalid min_total_ratio/, "min_total_ratio '$total' is rejected");
+ }
+ for my $balanced (-0.1, 1.5, 'abc', 'nan') {
+ eval { $planner->suggest_rebalance($movable, balanced_imbalance => $balanced) };
+ like($@, qr/invalid balanced_imbalance/, "balanced_imbalance '$balanced' is rejected");
+ }
+ for my $limit (-1, 1.5, 'abc') {
+ eval { $planner->suggest_rebalance($movable, busiest_nodes => $limit) };
+ like($@, qr/invalid busiest_nodes/, "busiest_nodes '$limit' is rejected");
+ }
+ for my $allowed ('node2', ['node2'], { 'vm:101' => 'node2' }) {
+ eval { $planner->suggest_rebalance($movable, allowed_targets => $allowed) };
+ like($@, qr/invalid allowed_targets/, 'malformed allowed_targets are rejected');
+ }
+ eval { $planner->suggest_rebalance($movable, method => 'bogus') };
+ like($@, qr/invalid method/, 'an unknown method is rejected');
+};
+
+subtest 'suggest_rebalance: no proposal for a balanced cluster with a negligible gain' => sub {
+ # a nearly balanced cluster whose slight imbalance comes from host overhead on node1, where
+ # moving its two tiny guests would only lower the imbalance a bit
+ my $usage =
+ sub { return { maxcpu => 2, maxmem => 4 * $GB, cpu => 0.02, mem => int(0.02 * $GB) } };
+ my $guest_stats = {
+ 'ct:101' => { node => 'node1', running => 1, usage => $usage->() },
+ 'ct:102' => { node => 'node1', running => 1, usage => $usage->() },
+ };
+ my $node_stats = {
+ node1 => { maxcpu => 8, cpu => 2.54, maxmem => 16 * $GB, mem => int(5.04 * $GB) },
+ node2 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ node3 => { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB },
+ };
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ my $unguarded = $planner->suggest_rebalance($movable, min_total_ratio => 0);
+ cmp_ok(scalar($unguarded->{moves}->@*), '>', 0, 'improving migrations exist');
+ my $gain = $unguarded->{imbalance}->{before} - $unguarded->{imbalance}->{after};
+ my $ratio = $gain / $unguarded->{imbalance}->{before};
+ cmp_ok(
+ $ratio,
+ '<',
+ PVE::ResourceScheduling::Planner::DEFAULT_MIN_TOTAL_RATIO,
+ 'but they lower the imbalance by less than the default ratio',
+ );
+
+ my $plan = $planner->suggest_rebalance($movable);
+ cmp_ok(
+ $plan->{imbalance}->{before},
+ '<',
+ PVE::ResourceScheduling::Planner::DEFAULT_BALANCED_IMBALANCE,
+ 'the cluster counts as balanced',
+ );
+ is_deeply($plan->{moves}, [], 'no migration is proposed');
+ is($plan->{imbalance}->{after}, $plan->{imbalance}->{before}, 'after equals before');
+
+ my $lower = $planner->suggest_rebalance($movable, min_total_ratio => $ratio / 2);
+ plan_is($lower, $unguarded, 'a lower ratio keeps the proposal');
+ plan_is(
+ $planner->suggest_rebalance($movable, balanced_imbalance => 0),
+ $unguarded,
+ 'a zero balanced imbalance disables the guard',
+ );
+};
+
+subtest 'suggest_rebalance: no proposal for moving a nearly idle guest' => sub {
+ # node1 is loaded by a single guest, which would overload any other node just as much, so
+ # only moving the nearly idle guest improves the imbalance at all
+ my $guest_stats = {
+ 'ct:100' => guest('node1', cpu => 2, mem => $GB),
+ 'ct:101' => guest('node2', cpu => 0.001, mem => int(0.01 * $GB)),
+ };
+ my $node_stats = {
+ node1 => { maxcpu => 4, cpu => 2.1, maxmem => 4 * $GB, mem => 3 * $GB },
+ node2 => { maxcpu => 4, cpu => 0.4, maxmem => 4 * $GB, mem => 3 * $GB },
+ node3 => { maxcpu => 4, cpu => 0.2, maxmem => 4 * $GB, mem => 3 * $GB },
+ };
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ my $plan = $planner->suggest_rebalance($movable);
+ cmp_ok(
+ $plan->{imbalance}->{before},
+ '>=',
+ PVE::ResourceScheduling::Planner::DEFAULT_BALANCED_IMBALANCE,
+ 'the cluster does not count as balanced',
+ );
+ is_deeply($plan->{moves}, [], 'no migration is proposed');
+ is($plan->{imbalance}->{after}, $plan->{imbalance}->{before}, 'after equals before');
+};
+
+subtest 'suggest_rebalance: a small gain is kept for an imbalanced cluster' => sub {
+ # node1 is overloaded by a guest that may not be moved, only two small guests may
+ my $small =
+ sub { return { maxcpu => 2, maxmem => 4 * $GB, cpu => 0.1, mem => int(0.1 * $GB) } };
+ my $guest_stats = {
+ 'vm:100' => guest('node1', cpu => 4, mem => 8 * $GB),
+ 'ct:101' => { node => 'node1', running => 1, usage => $small->() },
+ 'ct:102' => { node => 'node1', running => 1, usage => $small->() },
+ };
+ my $node_stats = {
+ node1 => { maxcpu => 8, cpu => 5.2, maxmem => 16 * $GB, mem => int(10.2 * $GB) },
+ node2 => { maxcpu => 8, cpu => 1, maxmem => 16 * $GB, mem => 2 * $GB },
+ node3 => { maxcpu => 8, cpu => 1, maxmem => 16 * $GB, mem => 2 * $GB },
+ };
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = ['ct:101', 'ct:102'];
+
+ my $plan = $planner->suggest_rebalance($movable);
+ cmp_ok(
+ $plan->{imbalance}->{before},
+ '>=',
+ PVE::ResourceScheduling::Planner::DEFAULT_BALANCED_IMBALANCE,
+ 'the cluster does not count as balanced',
+ );
+ cmp_ok(scalar($plan->{moves}->@*), '>', 0, 'the small guests are moved');
+ my ($before, $after) = $plan->{imbalance}->@{qw(before after)};
+ cmp_ok(
+ ($before - $after) / $before,
+ '<',
+ PVE::ResourceScheduling::Planner::DEFAULT_MIN_TOTAL_RATIO,
+ 'although the gain is small',
+ );
+ is_deeply(
+ $planner->suggest_rebalance($movable, balanced_imbalance => 1)->{moves},
+ [],
+ 'the same gain is dropped where the cluster counts as balanced',
+ );
+};
+
+subtest 'suggest_rebalance: the tail after a big step is cut' => sub {
+ # without overhead, the first step gains about 0.36 and the next two about 0.03 each
+ my $usage =
+ sub { return (maxcpu => 4, maxmem => 8 * $GB, cpu => $_[0], mem => $_[1] * $GB) };
+ my $guest_stats = {
+ 'vm:101' => guest('node3', $usage->(0.1, 1)),
+ 'vm:102' => guest('node1', $usage->(1, 0.25)),
+ 'vm:103' => guest('node1', $usage->(1, 2)),
+ 'vm:104' => guest('node3', $usage->(0.2, 2)),
+ 'vm:105' => guest('node3', $usage->(0.5, 6)),
+ };
+ my $node_stats = nodes(3);
+ for my $guest (values %$guest_stats) {
+ $node_stats->{ $guest->{node} }->{cpu} += $guest->{usage}->{cpu};
+ $node_stats->{ $guest->{node} }->{mem} += $guest->{usage}->{mem};
+ }
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ my $unguarded = $planner->suggest_rebalance($movable, min_step_ratio => 0);
+ my $first = $planner->suggest_rebalance($movable, min_step_ratio => 0, max_moves => 1);
+ my $first_gain = $first->{imbalance}->{before} - $first->{imbalance}->{after};
+ my $tail_gain = $first->{imbalance}->{after} - $unguarded->{imbalance}->{after};
+ cmp_ok(scalar($unguarded->{moves}->@*), '>', 1,
+ 'more improving steps follow the first one');
+ cmp_ok($tail_gain, '<', 0.2 * $first_gain, 'and gain little compared to it');
+
+ plan_is(
+ $planner->suggest_rebalance($movable),
+ $first,
+ 'the default stops after the big step',
+ );
+ plan_is(
+ $planner->suggest_rebalance($movable, min_step_ratio => 0.05),
+ $unguarded,
+ 'a lower step ratio keeps the tail',
+ );
+};
+
+subtest 'suggest_rebalance: many similar steps are kept' => sub {
+ my ($node_stats, $guest_stats) = synthetic_cluster(16, 1000);
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ my $plan = $planner->suggest_rebalance($movable);
+ is(
+ scalar($plan->{moves}->@*),
+ PVE::ResourceScheduling::Planner::DEFAULT_MAX_MOVES,
+ 'all steps up to the limit are taken',
+ );
+ cmp_ok(
+ $plan->{imbalance}->{after},
+ '<',
+ $plan->{imbalance}->{before},
+ 'the proposal improves',
+ );
+ plan_is(
+ $plan,
+ $planner->suggest_rebalance($movable, min_step_ratio => 0, min_total_ratio => 0),
+ 'the guards do not change the proposal',
+ );
+};
+
+subtest 'suggest_rebalance: relief for a single overloaded node in a larger cluster' => sub {
+ # One saturated node among half-loaded peers. Its share of the imbalance shrinks with the
+ # number of nodes, so the cluster counts as balanced, and with many small guests one proposal
+ # lowers the imbalance only a little in absolute terms, but the overloaded node needs relief.
+ for my $peers (8, 16, 32) {
+ my ($node_stats, $guest_stats) = ({}, {});
+ my $vmid = 100;
+ for my $i (0 .. $peers) {
+ my $node = sprintf("node%02d", $i + 1);
+ my $guests = $i == 0 ? 128 : 64;
+ $node_stats->{$node} =
+ { maxcpu => 128, cpu => $guests, maxmem => 512 * $GB, mem => 4 * $guests * $GB };
+ $guest_stats->{ 'vm:' . $vmid++ } = guest($node, cpu => 1, mem => 4 * $GB)
+ for 1 .. $guests;
+ }
+
+ my $plan = planner('dynamic', $node_stats, $guest_stats)
+ ->suggest_rebalance([sort keys %$guest_stats]);
+ my ($before, $after) = $plan->{imbalance}->@{qw(before after)};
+
+ is(scalar($plan->{moves}->@*), 10, "$peers peers: the default request takes all steps");
+ ok(
+ !(grep { $_->{source} ne 'node01' } $plan->{moves}->@*),
+ "$peers peers: all moves relieve the overloaded node",
+ );
+ cmp_ok(
+ $before - $after,
+ '>=',
+ PVE::ResourceScheduling::Planner::DEFAULT_MIN_TOTAL_RATIO * $before,
+ "$peers peers: the proposal lowers the imbalance noticeably",
+ );
+ }
+};
+
+subtest 'suggest_rebalance: restricted target nodes' => sub {
+ my $guest_stats = {};
+ $guest_stats->{"vm:10$_"} = guest('node1', cpu => 1, mem => 2 * $GB) for 0 .. 5;
+ my $node_stats = nodes(4);
+ $node_stats->{node1}->@{qw(cpu mem)} = (6, 12 * $GB);
+
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ for my $targets (['node4'], ['node1', 'node4'], ['node3', 'node4', 'nosuchnode']) {
+ my $name = join(',', @$targets);
+ my $wanted = { map { $_ => 1 } @$targets };
+ my $plan = $planner->suggest_rebalance($movable, targets => $targets);
+
+ cmp_ok(scalar($plan->{moves}->@*), '>', 0, "$name: guests are moved");
+ ok(
+ !(grep { !$wanted->{ $_->{target} } } $plan->{moves}->@*),
+ "$name: all moves go to a target node",
+ );
+ ok(
+ !(grep { $_->{source} ne 'node1' } $plan->{moves}->@*),
+ "$name: the overloaded node is drained, whether it is a target or not",
+ );
+ is(
+ $plan->{imbalance}->{before},
+ $planner->node_imbalance(),
+ "$name: before is the imbalance over all nodes",
+ );
+ my $replayed =
+ replay('dynamic', $node_stats, $guest_stats, $plan->{moves})->node_imbalance();
+ ok(
+ abs($replayed - $plan->{imbalance}->{after}) < $EPS,
+ "$name: after is the imbalance over all nodes",
+ );
+ }
+
+ # with only one target node, the idle third node stays idle and the cluster less balanced
+ cmp_ok(
+ $planner->suggest_rebalance($movable, targets => ['node4'])->{imbalance}->{after},
+ '>',
+ $planner->suggest_rebalance($movable)->{imbalance}->{after},
+ 'a single target node balances less than all nodes',
+ );
+
+ my $skipping = PVE::ResourceScheduling::Planner->new(
+ mode => 'dynamic',
+ nodes => [qw(node1 node2 node3 node4 node5)],
+ node_stats => { %$node_stats, node5 => { maxcpu => 0, maxmem => 0 } },
+ guest_stats => $guest_stats,
+ );
+ for my $targets ([], ['nosuchnode'], ['node5']) {
+ my $plan = $skipping->suggest_rebalance($movable, targets => $targets);
+ my $name = '[' . join(',', @$targets) . ']';
+ is_deeply($plan->{moves}, [], "$name: no usable target node, no moves");
+ is(
+ $plan->{imbalance}->{after},
+ $plan->{imbalance}->{before},
+ "$name: after equals before",
+ );
+ }
+
+ eval { $planner->suggest_rebalance($movable, targets => 'node4') };
+ like($@, qr/invalid targets/, 'targets must be an array reference');
+};
+
+subtest 'suggest_rebalance: a guest moved twice ends on a target node' => sub {
+ # without overhead and with node2 and node3 as targets, vm:104 first moves to node3 and in the
+ # third step on to node2, which gains about 0.02
+ my $usage =
+ sub { return (maxcpu => 4, maxmem => 32 * $GB, cpu => $_[0], mem => $_[1] * $GB) };
+ my $guest_stats = {
+ 'vm:101' => guest('node4', $usage->(4, 4)),
+ 'vm:102' => guest('node2', $usage->(2, 6)),
+ 'vm:103' => guest('node3', $usage->(1, 11)),
+ 'vm:104' => guest('node1', $usage->(2, 3)),
+ 'vm:105' => guest('node1', $usage->(7, 13)),
+ 'vm:106' => guest('node2', $usage->(1, 14)),
+ 'vm:107' => guest('node4', $usage->(5, 9)),
+ };
+ my $node_stats = nodes(4, maxcpu => 16, maxmem => 64 * $GB);
+ for my $guest (values %$guest_stats) {
+ $node_stats->{ $guest->{node} }->{cpu} += $guest->{usage}->{cpu};
+ $node_stats->{ $guest->{node} }->{mem} += $guest->{usage}->{mem};
+ }
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+
+ my $plan = sub {
+ my ($steps) = @_;
+ my $plan = $planner->suggest_rebalance(
+ [sort keys %$guest_stats],
+ targets => ['node2', 'node3'],
+ max_moves => $steps,
+ );
+ return { map { $_->{sid} => $_ } $plan->{moves}->@* }, $plan;
+ };
+
+ my ($two) = $plan->(2);
+ my ($three) = $plan->(3);
+ is_deeply(
+ $two->{'vm:104'},
+ { sid => 'vm:104', source => 'node1', target => 'node3' },
+ 'vm:104 first moves to node3',
+ );
+ is_deeply(
+ $three->{'vm:104'},
+ { sid => 'vm:104', source => 'node1', target => 'node2' },
+ 'and is then reported with a single move to node2',
+ );
+
+ my (undef, $full) = $plan->(PVE::ResourceScheduling::Planner::DEFAULT_MAX_MOVES);
+ ok(
+ !(grep { $_->{target} ne 'node2' && $_->{target} ne 'node3' } $full->{moves}->@*),
+ 'all collapsed moves end on a target node',
+ );
+ my $replayed =
+ replay('dynamic', $node_stats, $guest_stats, $full->{moves})->node_imbalance();
+ ok(abs($replayed - $full->{imbalance}->{after}) < $EPS,
+ 'after matches the collapsed moves');
+
+ # the same restriction given per guest yields the same proposals, double hop included
+ my $movable = [sort keys %$guest_stats];
+ my $per_guest = { map { $_ => ['node2', 'node3'] } @$movable };
+ for my $steps (2, 3, PVE::ResourceScheduling::Planner::DEFAULT_MAX_MOVES) {
+ plan_is(
+ $planner->suggest_rebalance(
+ $movable,
+ allowed_targets => $per_guest,
+ max_moves => $steps,
+ ),
+ $planner->suggest_rebalance(
+ $movable,
+ targets => ['node2', 'node3'],
+ max_moves => $steps,
+ ),
+ "$steps steps: a per-guest restriction acts like the same global one",
+ );
+ }
+};
+
+subtest 'suggest_rebalance: per-guest target nodes' => sub {
+ my $guest_stats = {};
+ $guest_stats->{"vm:10$_"} = guest('node1', cpu => 1, mem => 2 * $GB) for 0 .. 5;
+ my $node_stats = nodes(4);
+ $node_stats->{node1}->@{qw(cpu mem)} = (6, 12 * $GB);
+
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ my $allowed = {
+ 'vm:100' => ['node3'],
+ 'vm:101' => [], # unmovable
+ 'vm:102' => ['node1', 'nosuchnode'], # only its current node, unmovable
+ 'vm:103' => ['node2', 'node4'],
+ };
+ my $check = sub {
+ my ($name, $plan, $may_go) = @_;
+ my $moved = { map { $_->{sid} => $_->{target} } $plan->{moves}->@* };
+ ok(
+ !(grep { $may_go->{$_} && !$may_go->{$_}->{ $moved->{$_} } } keys %$moved),
+ "$name: restricted guests only move to their allowed nodes",
+ );
+ ok(!$moved->{'vm:101'} && !$moved->{'vm:102'}, "$name: guests without target stay");
+ my $replayed =
+ replay('dynamic', $node_stats, $guest_stats, $plan->{moves})->node_imbalance();
+ ok(
+ abs($replayed - $plan->{imbalance}->{after}) < $EPS,
+ "$name: after matches the moves",
+ );
+ return $moved;
+ };
+
+ my $plan = $planner->suggest_rebalance($movable, allowed_targets => $allowed);
+ my $moved = $check->(
+ 'unrestricted',
+ $plan,
+ {
+ 'vm:100' => { node3 => 1 },
+ 'vm:101' => {},
+ 'vm:102' => {},
+ 'vm:103' => { node2 => 1, node4 => 1 },
+ },
+ );
+ is($moved->{'vm:100'}, 'node3', 'a guest with one allowed node moves there');
+ cmp_ok(scalar(keys %$moved), '==', 4, 'the other guests balance the cluster');
+
+ my $both = $planner->suggest_rebalance(
+ $movable,
+ allowed_targets => $allowed,
+ targets => ['node4'],
+ );
+ $moved = $check->(
+ 'with global targets',
+ $both,
+ {
+ 'vm:100' => {},
+ 'vm:101' => {},
+ 'vm:102' => {},
+ 'vm:103' => { node4 => 1 },
+ map { ("vm:10$_" => { node4 => 1 }) } 4 .. 5,
+ },
+ );
+ ok(!$moved->{'vm:100'}, 'per-guest and global restrictions intersect');
+};
+
+subtest 'suggest_rebalance: candidate pruning' => sub {
+ # stacked guests, so the busiest node and the least loaded ones are the only sensible choices
+ my $guest_stats = {};
+ $guest_stats->{"vm:10$_"} = guest('node1', cpu => 1, mem => 2 * $GB) for 0 .. 5;
+ my $node_stats = nodes(4);
+ $node_stats->{node1}->@{qw(cpu mem)} = (6, 12 * $GB);
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ plan_is(
+ $planner->suggest_rebalance($movable, busiest_nodes => 1),
+ $planner->suggest_rebalance($movable, busiest_nodes => 0),
+ 'pruning to the busiest node yields the same proposal',
+ );
+
+ # a restricted guest gets the least loaded of its own allowed nodes, not none
+ is_deeply(
+ $planner->suggest_rebalance(
+ ['vm:100'],
+ busiest_nodes => 1,
+ allowed_targets => { 'vm:100' => ['node4'] },
+ )->{moves},
+ [{ sid => 'vm:100', source => 'node1', target => 'node4' }],
+ 'pruning picks among the allowed nodes of a restricted guest',
+ );
+
+ # The busiest node only holds a guest too big to move anywhere, so a step pruned to it finds
+ # nothing, while moving the small guests of the second busiest node helps.
+ my $fallback_guests = {
+ 'vm:100' => guest('node1', cpu => 6, mem => 12 * $GB),
+ map { ("ct:10$_" => guest('node2', cpu => 1, mem => 2 * $GB)) } 1 .. 4,
+ };
+ my $fallback_nodes = {
+ node1 => { maxcpu => 8, cpu => 6, maxmem => 16 * $GB, mem => 12 * $GB },
+ node2 => { maxcpu => 8, cpu => 4, maxmem => 16 * $GB, mem => 8 * $GB },
+ node3 => { maxcpu => 8, cpu => 0.8, maxmem => 16 * $GB, mem => int(1.6 * $GB) },
+ };
+ my $fallback = planner('dynamic', $fallback_nodes, $fallback_guests);
+ my $fallback_movable = [sort keys %$fallback_guests];
+ my $pruned = $fallback->suggest_rebalance($fallback_movable, busiest_nodes => 1);
+ cmp_ok(
+ scalar($pruned->{moves}->@*),
+ '>',
+ 0,
+ 'a pruned step without result is retried in full',
+ );
+ is($pruned->{moves}->[0]->{source}, 'node2', 'the retry finds the small guests');
+ plan_is(
+ $pruned,
+ $fallback->suggest_rebalance($fallback_movable, busiest_nodes => 0),
+ 'and yields the same proposal as without pruning',
+ );
+
+ # On nodes of different size, the best target need not be the least loaded node: moving a
+ # guest from the small, busy node1 to the small node3 beats the large but idle node2 here.
+ my $sized_guests = {};
+ $sized_guests->{"vm:10$_"} = guest('node1', cpu => 1, mem => 2 * $GB) for 0 .. 5;
+ my $sized_nodes = {
+ node1 => { maxcpu => 8, cpu => 6, maxmem => 16 * $GB, mem => 12 * $GB },
+ node2 => { maxcpu => 64, cpu => 3.2, maxmem => 128 * $GB, mem => int(6.4 * $GB) },
+ node3 => { maxcpu => 8, cpu => 2.4, maxmem => 16 * $GB, mem => int(4.8 * $GB) },
+ };
+ my $sized = planner('dynamic', $sized_nodes, $sized_guests);
+ my $sized_movable = [sort keys %$sized_guests];
+ my $sized_step =
+ $sized->suggest_rebalance($sized_movable, busiest_nodes => 1, max_moves => 1);
+ is(
+ $sized_step->{moves}->[0]->{target},
+ 'node3',
+ 'the busier node of another size is chosen',
+ );
+ my $sized_plan = $sized->suggest_rebalance($sized_movable, busiest_nodes => 1);
+ plan_is(
+ $sized_plan,
+ $sized->suggest_rebalance($sized_movable, busiest_nodes => 0),
+ 'pruning offers the least loaded node of every size and loses nothing',
+ );
+
+ # pruning takes effect with more nodes than its limit
+ my ($synth_nodes, $synth_guests) = synthetic_cluster(16, 300);
+ my $synth = planner('dynamic', $synth_nodes, $synth_guests);
+ my $plan = $synth->suggest_rebalance([sort keys %$synth_guests], busiest_nodes => 3);
+ cmp_ok(scalar($plan->{moves}->@*), '>', 0, 'a pruned proposal on a larger cluster');
+ my $replayed =
+ replay('dynamic', $synth_nodes, $synth_guests, $plan->{moves})->node_imbalance();
+ ok(abs($replayed - $plan->{imbalance}->{after}) < $EPS, 'after matches the pruned moves');
+ plan_is(
+ $plan,
+ $synth->suggest_rebalance([sort keys %$synth_guests], busiest_nodes => 0),
+ 'and is the same as without pruning',
+ );
+
+ # TOPSIS ranks the candidates relative to each other, so it always gets all of them
+ plan_is(
+ $synth->suggest_rebalance(
+ [sort keys %$synth_guests],
+ method => 'topsis',
+ busiest_nodes => 1,
+ ),
+ $synth->suggest_rebalance(
+ [sort keys %$synth_guests],
+ method => 'topsis',
+ busiest_nodes => 0,
+ ),
+ 'topsis proposals are not pruned',
+ );
+};
+
+subtest 'suggest_rebalance: guests kept apart' => sub {
+ # two busy nodes and an idle one; each of the two guests may move to node3 on its own, but
+ # they must not end up there together
+ my $node_stats = {
+ node1 => { maxcpu => 16, cpu => 14, maxmem => 64 * $GB, mem => 56 * $GB },
+ node2 => { maxcpu => 16, cpu => 14, maxmem => 64 * $GB, mem => 56 * $GB },
+ node3 => { maxcpu => 16, cpu => 0, maxmem => 64 * $GB, mem => 0 },
+ };
+ my $guest_stats = {
+ 'vm:101' => guest('node1', cpu => 4, mem => 16 * $GB),
+ 'vm:102' => guest('node2', cpu => 4, mem => 16 * $GB),
+ };
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+ my $allowed = { 'vm:101' => ['node3'], 'vm:102' => ['node3'] };
+ my $separate = { 'vm:101' => ['vm:102'], 'vm:102' => ['vm:101'] };
+
+ my $unseparated = $planner->suggest_rebalance($movable, allowed_targets => $allowed);
+ is_deeply(
+ [sort map { $_->{target} } $unseparated->{moves}->@*],
+ ['node3', 'node3'],
+ 'without separation, both guests move to the idle node',
+ );
+
+ for my $map ($separate, { 'vm:101' => ['vm:102'] }, { 'vm:102' => ['vm:101'] }) {
+ my $name = join(',', sort keys %$map) . ' listed';
+ my $plan = $planner->suggest_rebalance(
+ $movable,
+ allowed_targets => $allowed,
+ separate => $map,
+ );
+ is(scalar($plan->{moves}->@*), 1, "$name: only one of them moves to node3");
+ is_deeply(
+ shared_nodes($guest_stats, $plan->{moves}, $separate),
+ [],
+ "$name: kept apart",
+ );
+ }
+
+ my $free = $planner->suggest_rebalance($movable, separate => $separate);
+ cmp_ok(scalar($free->{moves}->@*), '>', 0, 'without node restrictions guests still move');
+ is_deeply(
+ shared_nodes($guest_stats, $free->{moves}, $separate),
+ [],
+ 'and are kept apart on any node',
+ );
+ my $replayed =
+ replay('dynamic', $node_stats, $guest_stats, $free->{moves})->node_imbalance();
+ ok(abs($replayed - $free->{imbalance}->{after}) < $EPS, 'after matches the moves');
+
+ # a partner that may not move itself still blocks its node
+ my $pinned = $planner->suggest_rebalance(
+ ['vm:101'],
+ separate => { 'vm:101' => ['vm:102'], 'vm:102' => ['vm:999'] },
+ targets => ['node2'],
+ );
+ is_deeply($pinned->{moves}, [], 'the node of an unmovable partner is never a target');
+
+ eval { $planner->suggest_rebalance($movable, separate => 'vm:102') };
+ like($@, qr/invalid separate/, 'separate must be a hash reference');
+ for my $bad (['vm:102'], { 'vm:101' => 'vm:102' }) {
+ eval { $planner->suggest_rebalance($movable, separate => $bad) };
+ like($@, qr/invalid separate/, 'separate must map to array references');
+ }
+};
+
+subtest 'suggest_rebalance: three guests kept apart from each other' => sub {
+ my $node_stats = nodes(4, maxcpu => 16, maxmem => 64 * $GB);
+ my $guest_stats = {};
+ for my $i (1 .. 3) {
+ my $node = "node$i";
+ $guest_stats->{"vm:10$i"} = guest($node, cpu => 4, mem => 16 * $GB);
+ $guest_stats->{"vm:20$i"} = guest($node, cpu => 10, mem => 40 * $GB); # unmovable load
+ $node_stats->{$node}->@{qw(cpu mem)} = (14, 56 * $GB);
+ }
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = ['vm:101', 'vm:102', 'vm:103'];
+
+ my $unseparated = $planner->suggest_rebalance($movable);
+ cmp_ok(
+ scalar(grep { $_->{target} eq 'node4' } $unseparated->{moves}->@*),
+ '>',
+ 1,
+ 'without separation, several guests move to the idle node',
+ );
+
+ my $mutual = {
+ 'vm:101' => ['vm:102', 'vm:103'],
+ 'vm:102' => ['vm:101', 'vm:103'],
+ 'vm:103' => ['vm:101', 'vm:102'],
+ };
+ my $one_sided = { 'vm:101' => ['vm:102', 'vm:103'], 'vm:102' => ['vm:103'] };
+ for my $map ($mutual, $one_sided) {
+ my $name = $map == $mutual ? 'mutual' : 'one-sided';
+ my $plan = $planner->suggest_rebalance($movable, separate => $map);
+ is(
+ scalar(grep { $_->{target} eq 'node4' } $plan->{moves}->@*),
+ 1,
+ "$name: only one of them moves to the idle node",
+ );
+ is_deeply(shared_nodes($guest_stats, $plan->{moves}, $mutual), [], "$name: kept apart");
+ }
+};
+
+subtest 'suggest_rebalance: separation with double hops, pruning and undone moves' => sub {
+ # the double hop scenario: vm:104 moves to node3 and then on to node2, where vm:102 is
+ my $usage =
+ sub { return (maxcpu => 4, maxmem => 32 * $GB, cpu => $_[0], mem => $_[1] * $GB) };
+ my $guest_stats = {
+ 'vm:101' => guest('node4', $usage->(4, 4)),
+ 'vm:102' => guest('node2', $usage->(2, 6)),
+ 'vm:103' => guest('node3', $usage->(1, 11)),
+ 'vm:104' => guest('node1', $usage->(2, 3)),
+ 'vm:105' => guest('node1', $usage->(7, 13)),
+ 'vm:106' => guest('node2', $usage->(1, 14)),
+ 'vm:107' => guest('node4', $usage->(5, 9)),
+ };
+ my $node_stats = nodes(4, maxcpu => 16, maxmem => 64 * $GB);
+ for my $guest (values %$guest_stats) {
+ $node_stats->{ $guest->{node} }->{cpu} += $guest->{usage}->{cpu};
+ $node_stats->{ $guest->{node} }->{mem} += $guest->{usage}->{mem};
+ }
+ my $planner = planner('dynamic', $node_stats, $guest_stats);
+ my $movable = [sort keys %$guest_stats];
+
+ my $hops = $planner->suggest_rebalance($movable, targets => ['node2', 'node3']);
+ my ($hop) = grep { $_->{sid} eq 'vm:104' } $hops->{moves}->@*;
+ is($hop->{target}, 'node2', 'unseparated, vm:104 ends next to vm:102');
+
+ my $separate = { 'vm:104' => ['vm:102'] };
+ my $plan = $planner->suggest_rebalance(
+ $movable,
+ targets => ['node2', 'node3'],
+ separate => $separate,
+ );
+ is_deeply(
+ shared_nodes($guest_stats, $plan->{moves}, $separate),
+ [],
+ 'separated, it does not',
+ );
+ my $replayed =
+ replay('dynamic', $node_stats, $guest_stats, $plan->{moves})->node_imbalance();
+ ok(abs($replayed - $plan->{imbalance}->{after}) < $EPS, 'after matches the moves');
+
+ # pruned and full scans honor the separation alike
+ my ($synth_nodes, $synth_guests) = synthetic_cluster(16, 300);
+ # pair guests of the busy node1 with guests elsewhere, which are apart to begin with
+ my @busy = grep { $synth_guests->{$_}->{node} eq 'node1' } sort keys %$synth_guests;
+ my @other = grep { $synth_guests->{$_}->{node} ne 'node1' } sort keys %$synth_guests;
+ my $pairs = { map { $busy[$_] => [$other[$_]] } 0 .. min($#busy, $#other) };
+ my $synth = planner('dynamic', $synth_nodes, $synth_guests);
+ my $pruned = $synth->suggest_rebalance(
+ [sort keys %$synth_guests],
+ separate => $pairs,
+ busiest_nodes => 3,
+ max_moves => 30,
+ );
+ cmp_ok(scalar($pruned->{moves}->@*), '>', 0, 'a pruned proposal with separated guests');
+ is_deeply(shared_nodes($synth_guests, $pruned->{moves}, $pairs), [], 'keeps them apart');
+ plan_is(
+ $pruned,
+ $synth->suggest_rebalance(
+ [sort keys %$synth_guests],
+ separate => $pairs,
+ busiest_nodes => 0,
+ max_moves => 30,
+ ),
+ 'and equals the unpruned proposal',
+ );
+
+ # an undone move restores the placement the separation is checked against
+ my $static_nodes = {
+ node1 => { maxcpu => 16, maxmem => 64 * $GB },
+ node2 => { maxcpu => 16, maxmem => 64 * $GB },
+ node3 => { maxcpu => 64, maxmem => 64 * $GB },
+ };
+ my $static_guests = {
+ 'ct:100' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 0, maxmem => 4 * $GB } },
+ 'ct:101' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 0, maxmem => 4 * $GB } },
+ 'vm:102' =>
+ { node => 'node2', running => 1, usage => { maxcpu => 4, maxmem => 8 * $GB } },
+ };
+ my $static_separate = { 'ct:100' => ['ct:101'] };
+ my $static = planner('static', $static_nodes, $static_guests)->suggest_rebalance(
+ [sort keys %$static_guests],
+ separate => $static_separate,
+ min_step_ratio => 0,
+ );
+ is_deeply(
+ shared_nodes($static_guests, $static->{moves}, $static_separate),
+ [],
+ 'separated guests stay apart around an undone move',
+ );
+ $replayed =
+ replay('static', $static_nodes, $static_guests, $static->{moves})->node_imbalance();
+ ok(abs($replayed - $static->{imbalance}->{after}) < $EPS, 'after matches the moves');
+};
+
+subtest 'static mode works end to end' => sub {
+ my $planner = planner(
+ 'static',
+ {
+ node1 => { maxcpu => 8, maxmem => 16 * $GB },
+ node2 => { maxcpu => 8, maxmem => 16 * $GB },
+ node3 => { maxcpu => 8, maxmem => 16 * $GB },
+ },
+ {
+ 'vm:100' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 2, maxmem => 4 * $GB } },
+ 'vm:101' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 2, maxmem => 4 * $GB } },
+ 'vm:102' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 2, maxmem => 4 * $GB } },
+ },
+ );
+
+ is($planner->mode(), 'static', 'static mode is reported');
+ cmp_ok($planner->node_imbalance(), '>', 0, 'static imbalance from configured usage');
+
+ my $plan = $planner->suggest_rebalance([qw(vm:100 vm:101 vm:102)]);
+ is(scalar($plan->{moves}->@*), 2, 'one guest moves to each of the other nodes');
+ ok($plan->{imbalance}->{after} < $EPS, 'static rebalance balances the cluster');
+};
+
+subtest 'a single-node cluster is balanced and has nothing to move' => sub {
+ my $planner = planner(
+ 'dynamic',
+ { node1 => { maxcpu => 8, cpu => 4, maxmem => 16 * $GB, mem => 8 * $GB } },
+ { 'vm:100' => guest('node1', cpu => 4, mem => 8 * $GB) },
+ );
+
+ is($planner->node_imbalance(), 0, 'single-node cluster has zero imbalance');
+ is_deeply(
+ $planner->rank_target_nodes('vm:100'),
+ [],
+ 'single-node cluster offers no migration targets',
+ );
+ is_deeply(
+ $planner->suggest_rebalance(['vm:100'])->{moves},
+ [],
+ 'single-node cluster has nothing to rebalance',
+ );
+};
+
+# Parity with driving PVE::ResourceScheduling::Scheduler by hand, the way the HA usage drivers do:
+# add every node, then every running guest on its node. Static mode must match exactly, and so must
+# dynamic mode as long as no node measures less than its guests, where the reconcile is a no-op.
+my sub hand_scheduler {
+ my ($mode, $node_stats, $guest_stats) = @_;
+
+ my $scheduler = PVE::ResourceScheduling::Scheduler->new($mode);
+ $scheduler->add_node($_, $node_stats->{$_}) for sort keys %$node_stats;
+ for my $sid (sort keys %$guest_stats) {
+ my $guest = $guest_stats->{$sid};
+ next if !$guest->{running};
+ $scheduler->add_resource(
+ $sid,
+ { stats => $guest->{usage}, running => 1, 'current-node' => $guest->{node} },
+ );
+ }
+
+ return $scheduler;
+}
+
+my @parity_scenarios = (
+ {
+ name => 'static, equal capacity, guests stacked on node1',
+ mode => 'static',
+ nodes => { map { ("node$_" => { maxcpu => 8, maxmem => 32 * $GB }) } 1 .. 3 },
+ guests => {
+ 'vm:101' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 2, maxmem => 4 * $GB } },
+ 'vm:102' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 2, maxmem => 4 * $GB } },
+ 'vm:103' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 4, maxmem => 8 * $GB } },
+ 'vm:104' =>
+ { node => 'node2', running => 1, usage => { maxcpu => 1, maxmem => 2 * $GB } },
+ 'vm:105' =>
+ { node => 'node2', running => 0, usage => { maxcpu => 2, maxmem => 4 * $GB } },
+ },
+ },
+ {
+ name => 'static, heterogeneous capacity, uneven spread',
+ mode => 'static',
+ nodes => {
+ node1 => { maxcpu => 16, maxmem => 64 * $GB },
+ node2 => { maxcpu => 8, maxmem => 32 * $GB },
+ node3 => { maxcpu => 4, maxmem => 16 * $GB },
+ },
+ guests => {
+ 'ct:201' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 4, maxmem => 8 * $GB } },
+ 'ct:202' =>
+ { node => 'node1', running => 1, usage => { maxcpu => 2, maxmem => 8 * $GB } },
+ 'vm:203' =>
+ { node => 'node2', running => 1, usage => { maxcpu => 4, maxmem => 16 * $GB } },
+ 'vm:204' =>
+ { node => 'node3', running => 1, usage => { maxcpu => 1, maxmem => 2 * $GB } },
+ 'vm:205' =>
+ { node => 'node3', running => 0, usage => { maxcpu => 8, maxmem => 32 * $GB } },
+ },
+ },
+ {
+ name => 'dynamic, no node measures less than its guests',
+ mode => 'dynamic',
+ nodes => {
+ node1 => { maxcpu => 16, cpu => 4.0, maxmem => 64 * $GB, mem => 28 * $GB },
+ node2 => { maxcpu => 16, cpu => 1.5, maxmem => 64 * $GB, mem => 6 * $GB },
+ node3 => { maxcpu => 16, cpu => 0.5, maxmem => 64 * $GB, mem => 2 * $GB },
+ },
+ guests => {
+ 'vm:401' =>
+ guest('node1', maxcpu => 8, cpu => 2.0, maxmem => 32 * $GB, mem => 8 * $GB),
+ 'vm:402' =>
+ guest('node1', maxcpu => 8, cpu => 1.0, maxmem => 32 * $GB, mem => 4 * $GB),
+ 'vm:403' =>
+ guest('node2', maxcpu => 8, cpu => 1.5, maxmem => 32 * $GB, mem => 6 * $GB),
+ 'vm:404' =>
+ guest('node3', maxcpu => 8, cpu => 0.5, maxmem => 32 * $GB, mem => 2 * $GB),
+ 'vm:405' => { guest('node3', maxcpu => 4, maxmem => 8 * $GB)->%*, running => 0 },
+ },
+ },
+);
+
+for my $scenario (@parity_scenarios) {
+ subtest "parity with a hand-driven scheduler: $scenario->{name}" => sub {
+ my ($mode, $node_stats, $guest_stats) = $scenario->@{qw(mode nodes guests)};
+
+ my $hand = hand_scheduler($mode, $node_stats, $guest_stats);
+ my $planner = planner($mode, $node_stats, $guest_stats);
+
+ ok(
+ abs($planner->node_imbalance() - $hand->calculate_node_imbalance()) < $EPS,
+ 'node imbalance matches',
+ );
+
+ for my $sid (sort keys %$guest_stats) {
+ my $guest = $guest_stats->{$sid};
+ my @others = grep { $_ ne $guest->{node} } sort keys %$node_stats;
+
+ my $ranked = $planner->rank_target_nodes($sid);
+ if ($guest->{running}) {
+ my $candidate = { leader => $sid, resources => [$sid], nodes => [@others] };
+ my $expected = {
+ map { $_->{migration}->{'target-node'} => $_->{imbalance} }
+ $hand->score_best_balancing_migration_candidates(
+ [$candidate], scalar(@others),
+ )->@*
+ };
+ is_deeply(
+ [sort map { $_->{node} } @$ranked],
+ [sort keys %$expected],
+ "$sid: migration targets match",
+ );
+ ok(
+ !(
+ grep { abs($_->{imbalance} - $expected->{ $_->{node} }) >= $EPS }
+ @$ranked
+ ),
+ "$sid: migration imbalances match",
+ );
+ } else {
+ my $allowed = { map { $_ => 1 } @others };
+ my $expected = [
+ map { $_->[0] }
+ sort { $b->[1] <=> $a->[1] || $a->[0] cmp $b->[0] }
+ grep { $allowed->{ $_->[0] } }
+ $hand->score_nodes_to_start($guest->{usage})->@*
+ ];
+ is_deeply(
+ [map { $_->{node} } @$ranked],
+ $expected,
+ "$sid: placement order matches",
+ );
+ }
+ }
+
+ my @candidates;
+ for my $sid (sort keys %$guest_stats) {
+ my $guest = $guest_stats->{$sid};
+ next if !$guest->{running};
+ my @targets = grep { $_ ne $guest->{node} } sort keys %$node_stats;
+ push @candidates, { leader => $sid, resources => [$sid], nodes => [@targets] };
+ }
+ my $best = $hand->score_best_balancing_migration_candidates(\@candidates, 1)->[0];
+ my $improves = $hand->calculate_node_imbalance() - $best->{imbalance} > $EPS;
+
+ # only the scheduler decision matters here, not whether a single step is worth proposing
+ my $plan = $planner->suggest_rebalance(
+ [sort keys %$guest_stats],
+ max_moves => 1,
+ min_total_ratio => 0,
+ );
+ if ($improves) {
+ my ($sid, $source, $target) =
+ $best->{migration}->@{qw(sid source-node target-node)};
+ is_deeply(
+ $plan->{moves},
+ [{ sid => $sid, source => $source, target => $target }],
+ 'the first step is the best migration',
+ );
+ ok(
+ abs($plan->{imbalance}->{after} - $best->{imbalance}) < $EPS,
+ 'the imbalance after the step is the predicted one',
+ );
+ } else {
+ is_deeply($plan->{moves}, [], 'no step when the best migration does not improve');
+ }
+ };
+}
+
+subtest 'build(): mode selection and argument checks' => sub {
+ my $calls = {};
+ my $dynamic_nodes = {
+ node1 => { maxcpu => 8, cpu => 1, maxmem => 16 * $GB, mem => 2 * $GB },
+ node2 => { maxcpu => 8, cpu => 1, maxmem => 16 * $GB, mem => 2 * $GB },
+ };
+ my $static_nodes = {
+ node1 => { maxcpu => 8, maxmem => 16 * $GB },
+ node2 => { maxcpu => 8, maxmem => 16 * $GB },
+ node3 => { maxcpu => 8, maxmem => 16 * $GB },
+ };
+ my $passed_resolvers;
+
+ # a stopped guest, whose placement ranking lists the other nodes a planner covers
+ my $stopped =
+ sub { return { 'vm:100' => { node => 'node1', running => 0, usage => $_[0] } } };
+ my $covered = sub {
+ return [map { $_->{node} } $_[0]->rank_target_nodes('vm:100')->@*];
+ };
+
+ no warnings 'redefine';
+ local *PVE::ResourceScheduling::Stats::get_dynamic_node_stats = sub {
+ $calls->{dynamic_nodes}++;
+ return { map { $_ => { $dynamic_nodes->{$_}->%* } } keys %$dynamic_nodes };
+ };
+ local *PVE::ResourceScheduling::Stats::get_dynamic_guest_stats = sub {
+ $calls->{dynamic_guests}++;
+ return $stopped->({ maxcpu => 1, cpu => 0, maxmem => $GB, mem => 0 });
+ };
+ local *PVE::ResourceScheduling::Stats::get_static_node_stats = sub {
+ $calls->{static_nodes}++;
+ return { map { $_ => { $static_nodes->{$_}->%* } } keys %$static_nodes };
+ };
+ local *PVE::ResourceScheduling::Stats::get_static_guest_stats = sub {
+ ($passed_resolvers) = @_;
+ $calls->{static_guests}++;
+ return $stopped->({ maxcpu => 1, maxmem => $GB });
+ };
+
+ my $resolvers = { vm => sub { {} }, ct => sub { {} } };
+ my $build = sub { PVE::ResourceScheduling::Planner->build(@_) };
+
+ eval { $build->(mode => 'bogus', nodes => ['node1'], resolvers => $resolvers) };
+ like($@, qr/invalid scheduling mode 'bogus'/, 'an unknown mode is rejected');
+
+ for my $mode (undef, 'auto', 'static') {
+ $calls = {};
+ eval { $build->(defined($mode) ? (mode => $mode) : (), nodes => ['node1']) };
+ my $name = $mode // 'default';
+ like($@, qr/needs a 'resolvers' argument/, "$name mode requires resolvers");
+ is_deeply($calls, {}, "$name mode checks resolvers before gathering stats");
+ }
+
+ eval { $build->(mode => 'dynamic', resolvers => $resolvers) };
+ like($@, qr/missing required 'nodes'/, 'nodes are required');
+
+ $calls = {};
+ my $planner = $build->(nodes => ['node1', 'node2'], resolvers => $resolvers);
+ is($planner->mode(), 'dynamic', 'auto uses dynamic stats when all nodes have them');
+ ok(!$calls->{static_nodes} && !$calls->{static_guests},
+ 'auto does not gather static stats');
+
+ $calls = {};
+ $planner = $build->(nodes => [qw(node1 node2 node3)], resolvers => $resolvers);
+ is($planner->mode(), 'static', 'auto falls back to static if a node lacks dynamic stats');
+ is_deeply($covered->($planner), [qw(node2 node3)], 'the static fallback covers all nodes');
+ is($passed_resolvers, $resolvers, 'the resolvers are passed to the static guest stats');
+
+ $dynamic_nodes->{node2}->{maxmem} = 0;
+ $planner = $build->(nodes => [qw(node1 node2)], resolvers => $resolvers);
+ is($planner->mode(), 'static', 'auto falls back to static if a node has unusable stats');
+ $dynamic_nodes->{node2}->{maxmem} = 16 * $GB;
+
+ $planner = $build->(mode => 'dynamic', nodes => [qw(node1 node2 node3)]);
+ is($planner->mode(), 'dynamic', 'explicit dynamic mode does not fall back');
+ is_deeply($covered->($planner), ['node2'], 'and skips the node without stats');
+
+ delete $static_nodes->{node3};
+ $planner =
+ $build->(mode => 'static', nodes => [qw(node1 node2 node3)], resolvers => $resolvers);
+ is($planner->mode(), 'static', 'explicit static mode');
+ is_deeply($covered->($planner), ['node2'], 'skips the node without static stats');
+
+ local *PVE::ResourceScheduling::Stats::get_dynamic_node_stats = sub { die "ipc failure\n" };
+ eval { $build->(nodes => ['node1'], resolvers => $resolvers) };
+ is($@, "ipc failure\n", 'errors while gathering stats propagate');
+};
+
+done_testing();
diff --git a/src/tests/resource-scheduling-scheduler-tests.pl b/src/tests/resource-scheduling-scheduler-tests.pl
new file mode 100644
index 0000000..951677b
--- /dev/null
+++ b/src/tests/resource-scheduling-scheduler-tests.pl
@@ -0,0 +1,230 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+use Test::More;
+
+use PVE::RS::ResourceScheduling::Static;
+use PVE::RS::ResourceScheduling::Dynamic;
+
+use PVE::ResourceScheduling::Scheduler;
+
+my $GB = 1024 * 1024 * 1024;
+
+# The Scheduler is a thin mode-dispatching forwarder, so test it against the raw PVE::RS backend:
+# for the same operations it must return exactly what the underlying scheduler returns. That pins
+# the static/dynamic method dispatch and the argument forwarding without re-testing the backend's
+# own placement math. Placement scores come back unordered, so compare them keyed by node.
+my sub scores_by_node {
+ my ($score_list) = @_;
+ return { map { $_->[0] => $_->[1] } @$score_list };
+}
+
+subtest 'new() validates the mode' => sub {
+ is(
+ PVE::ResourceScheduling::Scheduler->new('static')->mode(),
+ 'static',
+ 'static scheduler reports its mode',
+ );
+ is(
+ PVE::ResourceScheduling::Scheduler->new('dynamic')->mode(),
+ 'dynamic',
+ 'dynamic scheduler reports its mode',
+ );
+ eval { PVE::ResourceScheduling::Scheduler->new('bogus') };
+ like($@, qr/invalid scheduling mode/, 'an unknown mode dies');
+};
+
+subtest 'static scheduler forwards verbatim to the raw backend' => sub {
+ my $raw = PVE::RS::ResourceScheduling::Static->new();
+ my $sched = PVE::ResourceScheduling::Scheduler->new('static');
+
+ $raw->add_node('node1', 8, int(32 * $GB));
+ $raw->add_node('node2', 4, int(16 * $GB));
+ $sched->add_node('node1', { maxcpu => 8, maxmem => int(32 * $GB) });
+ $sched->add_node('node2', { maxcpu => 4, maxmem => int(16 * $GB) });
+
+ is_deeply(
+ [sort $sched->list_nodes()],
+ [qw(node1 node2)],
+ 'list_nodes returns a flat sorted list',
+ );
+ ok($sched->contains_node('node1'), 'contains_node forwards');
+
+ my $service = {
+ stats => { maxcpu => 4, maxmem => 16 * $GB },
+ running => 1,
+ 'current-node' => 'node1',
+ };
+ $raw->add_service('vm:100', $service);
+ $sched->add_resource('vm:100', $service);
+
+ my $usage = { maxcpu => 2, maxmem => 4 * $GB };
+ is_deeply(
+ scores_by_node($sched->score_nodes_to_start($usage)),
+ scores_by_node($raw->score_nodes_to_start_service($usage)),
+ 'score_nodes_to_start dispatches to the raw score_nodes_to_start_service',
+ );
+ is(
+ $sched->calculate_node_imbalance(),
+ $raw->calculate_node_imbalance(),
+ 'calculate_node_imbalance forwards',
+ );
+
+ my $candidate = { leader => 'vm:100', resources => ['vm:100'], nodes => ['node2'] };
+ is_deeply(
+ $sched->score_best_balancing_migration_candidates([$candidate], 1),
+ $raw->score_best_balancing_migration_candidates([$candidate], 1),
+ 'score_best_balancing_migration_candidates forwards (bruteforce, static)',
+ );
+
+ $raw->remove_service_usage('vm:100');
+ $sched->remove_resource('vm:100');
+ is(
+ $sched->calculate_node_imbalance(),
+ $raw->calculate_node_imbalance(),
+ 'remove_resource forwards to the raw remove_service_usage (static)',
+ );
+
+ $sched->remove_node('node2');
+ ok(!$sched->contains_node('node2'), 'remove_node forwards');
+};
+
+subtest 'dynamic scheduler forwards verbatim to the raw backend' => sub {
+ my $node = sub { return { maxcpu => 8, cpu => $_[0], maxmem => 16 * $GB, mem => $_[1] } };
+
+ my $raw = PVE::RS::ResourceScheduling::Dynamic->new();
+ my $sched = PVE::ResourceScheduling::Scheduler->new('dynamic');
+ $raw->add_node('node1', $node->(6, 12 * $GB));
+ $raw->add_node('node2', $node->(0, 0));
+ $sched->add_node('node1', $node->(6, 12 * $GB));
+ $sched->add_node('node2', $node->(0, 0));
+
+ my $resource = {
+ stats => { maxcpu => 8, cpu => 3, maxmem => 16 * $GB, mem => 6 * $GB },
+ running => 1,
+ 'current-node' => 'node1',
+ };
+ $raw->add_resource('vm:200', $resource);
+ $sched->add_resource('vm:200', $resource);
+
+ my $usage = { maxcpu => 8, cpu => 2, maxmem => 16 * $GB, mem => 4 * $GB };
+ is_deeply(
+ scores_by_node($sched->score_nodes_to_start($usage)),
+ scores_by_node($raw->score_nodes_to_start_resource($usage)),
+ 'score_nodes_to_start dispatches to the raw score_nodes_to_start_resource',
+ );
+
+ my $candidate = { leader => 'vm:200', resources => ['vm:200'], nodes => ['node2'] };
+ is_deeply(
+ $sched->score_best_balancing_migration_candidates([$candidate], 1),
+ $raw->score_best_balancing_migration_candidates([$candidate], 1),
+ 'bruteforce migration scorer forwards',
+ );
+
+ $raw->remove_resource('vm:200');
+ $sched->remove_resource('vm:200');
+ is(
+ $sched->calculate_node_imbalance(),
+ $raw->calculate_node_imbalance(),
+ 'remove_resource forwards to the raw remove_resource (dynamic)',
+ );
+};
+
+subtest "the 'topsis' method arg selects the raw topsis scorer" => sub {
+ # three guests on a loaded node where the two scorers prefer different migrations, so a wrong
+ # dispatch cannot go unnoticed
+ my $raw = PVE::RS::ResourceScheduling::Dynamic->new();
+ my $sched = PVE::ResourceScheduling::Scheduler->new('dynamic');
+
+ my $nodes = { node1 => [8, 12], node2 => [6, 5], node3 => [4, 0] };
+ for my $node (sort keys %$nodes) {
+ my ($cpu, $mem) = $nodes->{$node}->@*;
+ my $stats = { maxcpu => 8, cpu => $cpu, maxmem => 16 * $GB, mem => $mem * $GB };
+ $raw->add_node($node, $stats);
+ $sched->add_node($node, $stats);
+ }
+
+ my $guests = { 'vm:101' => [3, 1], 'vm:102' => [0, 6], 'vm:103' => [3, 3] };
+ my @candidates;
+ for my $sid (sort keys %$guests) {
+ my ($cpu, $mem) = $guests->{$sid}->@*;
+ my $resource = {
+ stats => { maxcpu => 4, cpu => $cpu, maxmem => 8 * $GB, mem => $mem * $GB },
+ running => 1,
+ 'current-node' => 'node1',
+ };
+ $raw->add_resource($sid, $resource);
+ $sched->add_resource($sid, $resource);
+ push @candidates, { leader => $sid, resources => [$sid], nodes => ['node2', 'node3'] };
+ }
+
+ my $topsis = $raw->score_best_balancing_migration_candidates_topsis(\@candidates, 1);
+ my $bruteforce = $raw->score_best_balancing_migration_candidates(\@candidates, 1);
+ isnt(
+ $topsis->[0]->{migration}->{sid},
+ $bruteforce->[0]->{migration}->{sid},
+ 'the scorers disagree on this cluster',
+ );
+
+ is_deeply(
+ $sched->score_best_balancing_migration_candidates(\@candidates, 1, 'topsis'),
+ $topsis,
+ "'topsis' dispatches to the raw topsis scorer",
+ );
+ is_deeply(
+ $sched->score_best_balancing_migration_candidates(\@candidates, 1),
+ $bruteforce,
+ 'the default dispatches to the raw bruteforce scorer',
+ );
+};
+
+subtest 'static add_node accepts fractional capacities like the raw backend given int()' => sub {
+ # Static capacities can come from derived values. The wrapper coerces them to integers itself
+ # instead of relying on how the binding converts a fractional value; the result must match
+ # passing explicit integers to the raw backend.
+ my $raw = PVE::RS::ResourceScheduling::Static->new();
+ my $sched = PVE::ResourceScheduling::Scheduler->new('static');
+
+ eval {
+ $sched->add_node('node1', { maxcpu => 8.7, maxmem => 16.9 * $GB });
+ $sched->add_node('node2', { maxcpu => 8, maxmem => 16 * $GB });
+ };
+ ok(!$@, 'static add_node accepts fractional capacities') or diag $@;
+
+ $raw->add_node('node1', int(8.7), int(16.9 * $GB));
+ $raw->add_node('node2', 8, int(16 * $GB));
+
+ my %resources = (
+ 'vm:100' => { maxcpu => 4, maxmem => 4 * $GB, node => 'node1' },
+ 'vm:101' => { maxcpu => 2, maxmem => 2 * $GB, node => 'node2' },
+ );
+ for my $sid (sort keys %resources) {
+ my $r = $resources{$sid};
+ my $service = {
+ stats => { maxcpu => $r->{maxcpu}, maxmem => $r->{maxmem} },
+ running => 1,
+ 'current-node' => $r->{node},
+ };
+ $raw->add_service($sid, $service);
+ $sched->add_resource($sid, $service);
+ }
+
+ cmp_ok($raw->calculate_node_imbalance(), '>', 0, 'both nodes carry load');
+ is(
+ $sched->calculate_node_imbalance(),
+ $raw->calculate_node_imbalance(),
+ 'imbalance matches the raw backend given integer capacities',
+ );
+};
+
+subtest 'an unknown scoring method is rejected' => sub {
+ my $sched = PVE::ResourceScheduling::Scheduler->new('dynamic');
+ eval { $sched->score_best_balancing_migration_candidates([], 1, 'bogus') };
+ like($@, qr/invalid scoring method/, 'an unknown scoring method dies');
+};
+
+done_testing();
diff --git a/src/tests/resource-scheduling-stats-tests.pl b/src/tests/resource-scheduling-stats-tests.pl
new file mode 100644
index 0000000..48fbcac
--- /dev/null
+++ b/src/tests/resource-scheduling-stats-tests.pl
@@ -0,0 +1,260 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+use Test::More;
+
+use PVE::Cluster;
+use PVE::ResourceScheduling::Stats;
+
+subtest 'dynamic guest stats: RRD-derived usage and guest-type mapping' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return {
+ ids => {
+ 100 => { node => 'node1', type => 'qemu' },
+ 200 => { node => 'node2', type => 'lxc' },
+ 300 => { node => 'node1', type => 'storage' }, # not a guest
+ },
+ };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ return {
+ # index: 2 status, 5 maxcpu, 6 cpu-fraction, 7 maxmem, 8 mem
+ 'pve-vm-9.0/100' =>
+ [undef, undef, 'running', undef, undef, 4, 0.5, 8_000_000_000, 4_000_000_000],
+ 'pve-vm-9.0/200' =>
+ [undef, undef, 'stopped', undef, undef, 2, 0.0, 4_000_000_000, 1_000_000_000],
+ };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_dynamic_guest_stats();
+
+ is_deeply(
+ [sort keys %$stats],
+ ['ct:200', 'vm:100'],
+ 'maps guest types and skips non-guests',
+ );
+ is($stats->{'vm:100'}->{running}, 1, 'running guest flagged as running');
+ is($stats->{'ct:200'}->{running}, 0, 'stopped guest not flagged as running');
+
+ my $usage = $stats->{'vm:100'}->{usage};
+ is($usage->{maxcpu}, 4, 'dynamic maxcpu taken from RRD');
+ is($usage->{cpu}, 2, 'dynamic cpu is fraction times maxcpu');
+ is($usage->{maxmem}, 8_000_000_000, 'dynamic maxmem taken from RRD');
+ is($usage->{mem}, 4_000_000_000, 'dynamic mem taken from RRD');
+};
+
+subtest 'dynamic guest stats: run state and usage come from a single RRD dump' => sub {
+ my $dumps = 0;
+
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return { ids => { 100 => { node => 'node1', type => 'qemu' } } };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ $dumps++;
+ return {
+ 'pve-vm-9.0/100' =>
+ [undef, undef, 'running', undef, undef, 4, 0.5, 8_000_000_000, 4_000_000_000],
+ };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_dynamic_guest_stats();
+
+ is($dumps, 1, 'the RRD data is dumped once');
+ is($stats->{'vm:100'}->{running}, 1, 'run state taken from the dump');
+ is($stats->{'vm:100'}->{usage}->{cpu}, 2, 'usage taken from the same dump');
+};
+
+subtest 'guest run state: only the QMP running state counts as running' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return {
+ ids => {
+ 100 => { node => 'node1', type => 'qemu' },
+ 101 => { node => 'node1', type => 'qemu' },
+ 102 => { node => 'node1', type => 'qemu' },
+ },
+ };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ return {
+ 'pve-vm-9.0/100' => [undef, undef, 'running'],
+ 'pve-vm-9.0/101' => [undef, undef, 'paused'],
+ # 102 has no RRD entry at all, e.g. never started since its creation
+ };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_cluster_guest_stats();
+
+ is($stats->{'vm:100'}->{state}, 'started', 'running guest is started');
+ is($stats->{'vm:101'}->{running}, 0, 'paused guest is not running');
+ is($stats->{'vm:101'}->{state}, 'stopped', 'paused guest is treated as stopped');
+ is($stats->{'vm:102'}->{running}, 0, 'guest without RRD entry is not running');
+};
+
+subtest 'dynamic guest stats: VMs account host memory, containers fall back to mem' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return {
+ ids => {
+ 100 => { node => 'node1', type => 'qemu' },
+ 200 => { node => 'node1', type => 'lxc' },
+ },
+ };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ my @vm = (undef, undef, 'running', undef, undef, 4, 0.5, 8_000_000_000, 4_000_000_000);
+ my @ct = (undef, undef, 'running', undef, undef, 2, 0.5, 4_000_000_000, 1_000_000_000);
+ # index 15 is memhost, which pvestatd records as unknown for containers
+ return {
+ 'pve-vm-9.0/100' => [@vm, (undef) x 6, 5_000_000_000],
+ 'pve-vm-9.0/200' => [@ct, (undef) x 7],
+ };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_dynamic_guest_stats();
+
+ is($stats->{'vm:100'}->{usage}->{mem}, 5_000_000_000, 'VM mem is its host memory usage');
+ is($stats->{'ct:200'}->{usage}->{mem}, 1_000_000_000, 'CT mem falls back to memory usage');
+};
+
+subtest 'dynamic guest stats: a reported zero host memory usage is kept' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return { ids => { 100 => { node => 'node1', type => 'qemu' } } };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ my @vm = (undef, undef, 'running', undef, undef, 4, 0.5, 8_000_000_000, 4_000_000_000);
+ return { 'pve-vm-9.0/100' => [@vm, (undef) x 6, 0] };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_dynamic_guest_stats();
+
+ # only an unknown host memory usage falls back, a known zero is a value like any other
+ is($stats->{'vm:100'}->{usage}->{mem}, 0, 'zero host memory usage does not fall back');
+};
+
+subtest 'cluster guest enumeration returns a skeleton with empty usage' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return { ids => { 100 => { node => 'node1', type => 'qemu' } } };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ return { 'pve-vm-9.0/100' => [undef, undef, 'running'] };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_cluster_guest_stats();
+
+ is_deeply(
+ $stats->{'vm:100'},
+ {
+ id => 100,
+ node => 'node1',
+ type => 'vm',
+ state => 'started',
+ running => 1,
+ usage => {},
+ },
+ 'enumeration returns guest skeleton with empty usage',
+ );
+};
+
+subtest 'dynamic node stats: only node RRD keys, cpu scaled by core count' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::rrd_dump = sub {
+ return {
+ 'pve-node-9.0/node1' =>
+ [undef, undef, undef, undef, 8, 0.25, undef, 16_000_000_000, 8_000_000_000],
+ 'pve-node-9.0/node2' =>
+ [undef, undef, undef, undef, 8, 0.0, undef, 16_000_000_000, 2_000_000_000],
+ 'pve-vm-9.0/100' => [undef, undef, 'running'], # must be ignored
+ 'some-unrelated-key' => [1, 2, 3],
+ };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_dynamic_node_stats();
+
+ is_deeply([sort keys %$stats], ['node1', 'node2'], 'only node RRD entries are returned');
+ is($stats->{node1}->{maxcpu}, 8, 'node maxcpu from RRD');
+ is($stats->{node1}->{cpu}, 2, 'node cpu is fraction times core count');
+ is($stats->{node1}->{maxmem}, 16_000_000_000, 'node maxmem from RRD');
+ is($stats->{node1}->{mem}, 8_000_000_000, 'node mem from RRD');
+};
+
+subtest 'static guest stats: usage comes from the given type resolver' => sub {
+ my $resolvers = {
+ vm => sub {
+ my ($conf) = @_;
+ return {
+ maxcpu => 0.0 + ($conf->{cores} // 1),
+ maxmem => int(($conf->{memory} // 512) * 1024 * 1024),
+ };
+ },
+ };
+
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return { ids => { 100 => { node => 'node1', type => 'qemu' } } };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ return { 'pve-vm-9.0/100' => [undef, undef, 'running'] };
+ };
+ local *PVE::Cluster::get_guest_config_properties = sub {
+ return { 100 => { cores => 4, memory => 2048 } };
+ };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_static_guest_stats($resolvers);
+
+ is($stats->{'vm:100'}->{usage}->{maxcpu}, 4, 'static maxcpu from resolver');
+ is($stats->{'vm:100'}->{usage}->{maxmem}, 2048 * 1024 * 1024,
+ 'static maxmem from resolver');
+};
+
+subtest 'static guest stats: a guest type without a resolver is an error' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_vmlist = sub {
+ return { ids => { 200 => { node => 'node1', type => 'lxc' } } };
+ };
+ local *PVE::Cluster::rrd_dump = sub {
+ return { 'pve-vm-9.0/200' => [undef, undef, 'running'] };
+ };
+ local *PVE::Cluster::get_guest_config_properties = sub { return {} };
+
+ # a resolvers map that lacks the 'ct' the guest needs
+ eval {
+ PVE::ResourceScheduling::Stats::get_static_guest_stats({ vm => sub { {} } });
+ };
+ like(
+ $@,
+ qr/no static stats resolver for guest type 'ct'/,
+ 'static stats without a matching resolver is rejected',
+ );
+};
+
+subtest 'static node stats: decoded from the static-info node kv' => sub {
+ no warnings 'redefine';
+ local *PVE::Cluster::get_node_kv = sub {
+ return { node1 => '{"cpus":8,"memory":16000000000}', node2 => 'garbage' };
+ };
+
+ my @warnings;
+ local $SIG{__WARN__} = sub { push @warnings, $_[0] };
+
+ my $stats = PVE::ResourceScheduling::Stats::get_static_node_stats();
+
+ is_deeply(
+ $stats->{node1},
+ { maxcpu => 8, maxmem => 16000000000 },
+ 'static node stats decoded from static-info kv',
+ );
+ ok(exists($stats->{node2}) && !defined($stats->{node2}), 'undecodable node info is undef');
+ like($warnings[0], qr/unable to decode static node info for 'node2'/,
+ 'decode error warned');
+};
+
+done_testing();
--
2.47.3
next prev parent reply other threads:[~2026-11-18 12:10 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 11:05 [PATCH guest-common,ha-manager,manager,docs 0/15] cluster resource scheduling for all guests Thomas Lamprecht
2026-11-18 11:05 ` Thomas Lamprecht [this message]
2026-11-18 11:05 ` [PATCH ha-manager 02/15] helpers: block both locations of a migrating negative affinity partner Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 03/15] usage: static: count services per node again for scoring fallback Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 04/15] config: allow getting motion info of many resources at once Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 05/15] env: pve2: delegate cluster stat gathering to scheduling library Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 06/15] usage: share one scheduler base between static and dynamic Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 07/15] ui: bulk action: translate error title and CT restart warning Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 08/15] ui: bulk migrate: show CT restart warning for cluster-wide migration Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 09/15] ui: migrate: recheck preconditions if the target changed meanwhile Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 10/15] api: cluster: add resource scheduling endpoints Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 11/15] ui: migrate: sort target nodes by scheduler recommendation Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 12/15] api: bulk migrate: allow a per-guest target node mapping Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 13/15] ui: bulk migrate: support per-guest target nodes Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 14/15] ui: dc: add a Rebalance Guests bulk action Thomas Lamprecht
2026-11-18 11:05 ` [PATCH docs 15/15] pvecm: document balancing the cluster load with migrations Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118110705.595989-2-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox