public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH manager 10/15] api: cluster: add resource scheduling endpoints
Date: Wed, 18 Nov 2026 12:05:17 +0100	[thread overview]
Message-ID: <20261118110705.595989-11-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118110705.595989-1-t.lamprecht@proxmox.com>

The cluster resource scheduler, which HA uses to place and rebalance
its resources, is not available for guests that HA does not manage.
Add two endpoints based on the PVE::ResourceScheduling library from
libpve-guest-common-perl, so the web interface can use it for all
guests.

recommend-target-nodes ranks the nodes as migration targets for a
guest by the cluster load imbalance the migration would result in.

rebalance-plan proposes up to a configurable number of migrations that
lower the cluster load imbalance, optionally only onto a set of target
nodes, for example to fill newly added nodes. Locked guests cannot be
migrated right now and are not proposed. It is a POST request, as the
list of guests to consider can exceed the request line limit of GET.

Guests managed by HA are only proposed if HA would carry out their
migration, that is, if their HA resource should be started and currently
is, and HA is not disarmed. They are only proposed for nodes that HA
considers online and their HA rules allow, as HA does not carry out
other migrations. Guests in a negative resource affinity are kept apart
across the whole proposal, as checking each move against the current
placement alone could still put them onto the same node. An HA guest
that HA would migrate together with other resources due to a positive
resource affinity is not proposed at all, as a per-guest proposal cannot
represent such a group move. Guests that are not proposed still count
towards the load of their node.

Both use the measured load and fall back to the configured guest
resources if a node did not report its load yet. Nodes in HA
maintenance mode are not considered, as they are about to be drained.
Only the load is considered; whether a migration is possible, for
example with regard to local resources, is still determined by the
existing migration precondition checks.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
NOTE: needs bumped version requirements for libpve-guest-common-perl
and pve-ha-manager, for their first releases with this series.

 PVE/API2/Cluster.pm                    |   7 +
 PVE/API2/Cluster/Makefile              |   3 +-
 PVE/API2/Cluster/ResourceScheduling.pm | 413 ++++++++++++++++++++++
 test/Makefile                          |   6 +-
 test/ResourceScheduling_test.pl        | 454 +++++++++++++++++++++++++
 5 files changed, 881 insertions(+), 2 deletions(-)
 create mode 100644 PVE/API2/Cluster/ResourceScheduling.pm
 create mode 100755 test/ResourceScheduling_test.pl

diff --git a/PVE/API2/Cluster.pm b/PVE/API2/Cluster.pm
index c487980a4..39b0eaf24 100644
--- a/PVE/API2/Cluster.pm
+++ b/PVE/API2/Cluster.pm
@@ -33,6 +33,7 @@ use PVE::API2::Cluster::Mapping;
 use PVE::API2::Cluster::MetricServer;
 use PVE::API2::Cluster::Notifications;
 use PVE::API2::Cluster::Qemu;
+use PVE::API2::Cluster::ResourceScheduling;
 use PVE::API2::ClusterConfig;
 use PVE::API2::Firewall::Cluster;
 use PVE::API2::HAConfig;
@@ -116,6 +117,11 @@ __PACKAGE__->register_method({
     path => 'bulk-action',
 });
 
+__PACKAGE__->register_method({
+    subclass => "PVE::API2::Cluster::ResourceScheduling",
+    path => 'resource-scheduling',
+});
+
 if ($have_sdn) {
     __PACKAGE__->register_method({
         subclass => "PVE::API2::Network::SDN",
@@ -175,6 +181,7 @@ __PACKAGE__->register_method({
             { name => 'options' },
             { name => 'qemu' },
             { name => 'replication' },
+            { name => 'resource-scheduling' },
             { name => 'resources' },
             { name => 'status' },
             { name => 'tasks' },
diff --git a/PVE/API2/Cluster/Makefile b/PVE/API2/Cluster/Makefile
index d3a56830c..83a407931 100644
--- a/PVE/API2/Cluster/Makefile
+++ b/PVE/API2/Cluster/Makefile
@@ -14,7 +14,8 @@ PERLSOURCE= 			\
 	Mapping.pm			\
 	MetricServer.pm		\
 	Notifications.pm	\
-	Qemu.pm
+	Qemu.pm				\
+	ResourceScheduling.pm
 
 all:
 
diff --git a/PVE/API2/Cluster/ResourceScheduling.pm b/PVE/API2/Cluster/ResourceScheduling.pm
new file mode 100644
index 000000000..428b9770a
--- /dev/null
+++ b/PVE/API2/Cluster/ResourceScheduling.pm
@@ -0,0 +1,413 @@
+package PVE::API2::Cluster::ResourceScheduling;
+
+use v5.36;
+
+use PVE::Cluster;
+use PVE::Exception qw(raise_param_exc raise_perm_exc);
+use PVE::HA::Config;
+use PVE::INotify;
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::LXC::Config;
+use PVE::QemuConfig;
+use PVE::RPCEnvironment;
+use PVE::Tools;
+
+use PVE::ResourceScheduling::Planner;
+
+use base qw(PVE::RESTHandler);
+
+# guest types as reported by the cluster guest list, mapped to the scheduler's resource id prefix
+my $guest_type_map = {
+    qemu => 'vm',
+    lxc => 'ct',
+};
+
+my $guest_type_by_prefix = { reverse %$guest_type_map };
+
+# The static (config-derived) per-guest-type usage resolvers the planner needs in static mode; the
+# HA daemons build their own from the HA resource plugins.
+my sub static_stats_resolvers {
+    return {
+        vm => sub($conf) {
+            return {
+                maxcpu => 0.0 + PVE::QemuConfig->get_derived_property($conf, 'max-cpu'),
+                maxmem => int(PVE::QemuConfig->get_derived_property($conf, 'max-memory')),
+            };
+        },
+        ct => sub($conf) {
+            return {
+                maxcpu => 0.0 + PVE::LXC::Config->get_derived_property($conf, 'max-cpu'),
+                maxmem => int(PVE::LXC::Config->get_derived_property($conf, 'max-memory')),
+            };
+        },
+    };
+}
+
+# The nodes that can currently take guests. A node in HA maintenance mode is about to be drained,
+# and as it is usually the least loaded node it would otherwise be the top recommendation.
+my sub schedulable_nodes($ha_status = PVE::HA::Config::read_manager_status()) {
+    my $members = PVE::Cluster::get_members() // {};
+    my $online = { map { $_ => 1 } grep { $members->{$_}->{online} } keys %$members };
+    # the local node serving this request is always up, even if pmxcfs has not (yet) broadcast its
+    # own membership, e.g. right after a restart; this also covers a standalone, non-clustered host
+    $online->{ PVE::INotify::nodename() } = 1;
+
+    my $ha_node_status = $ha_status->{node_status} // {};
+    delete $online->{$_} for grep { ($ha_node_status->{$_} // '') eq 'maintenance' } keys %$online;
+
+    return [sort keys %$online];
+}
+
+my sub guest_sid($vmid, $param_name) {
+    my $entry = (PVE::Cluster::get_vmlist()->{ids} // {})->{$vmid}
+        or raise_param_exc({ $param_name => "guest '$vmid' does not exist" });
+
+    my $type = $guest_type_map->{ $entry->{type} // '' }
+        or raise_param_exc(
+            { $param_name => "guest '$vmid' has unhandled type '" . ($entry->{type} // '') . "'" });
+
+    return "$type:$vmid";
+}
+
+my $mode_property = {
+    type => 'string',
+    enum => ['auto', 'dynamic', 'static'],
+    default => 'auto',
+    optional => 1,
+    description => "Usage information to base the result on. 'dynamic' uses the measured load,"
+        . " 'static' the configured guest resources and node capacity, and 'auto' uses 'dynamic'"
+        . " if the measured load of every node is available, else 'static'.",
+};
+
+my $mode_return_property = {
+    type => 'string',
+    enum => ['dynamic', 'static'],
+    description => "The usage information actually used.",
+};
+
+__PACKAGE__->register_method({
+    name => 'index',
+    path => '',
+    method => 'GET',
+    description => "Resource scheduling index.",
+    permissions => { user => 'all' },
+    parameters => {
+        additionalProperties => 0,
+        properties => {},
+    },
+    returns => {
+        type => 'array',
+        items => {
+            type => 'object',
+            properties => {},
+        },
+        links => [{ rel => 'child', href => "{name}" }],
+    },
+    code => sub($param) {
+        return [
+            { name => 'rebalance-plan' }, { name => 'recommend-target-nodes' },
+        ];
+    },
+});
+
+__PACKAGE__->register_method({
+    name => 'recommend_target_nodes',
+    path => 'recommend-target-nodes',
+    method => 'GET',
+    description => "Rank the cluster nodes as migration targets for a guest, best first. Only"
+        . " the load is considered; whether a migration to a node is possible, for example with"
+        . " regard to local resources or HA rules, must be checked separately.",
+    permissions => {
+        check => ['perm', '/vms/{vmid}', ['VM.Migrate']],
+    },
+    parameters => {
+        additionalProperties => 0,
+        properties => {
+            vmid => get_standard_option('pve-vmid'),
+            mode => $mode_property,
+            'target-nodes' => get_standard_option(
+                'pve-node-list',
+                {
+                    optional => 1,
+                    description => "Restrict the ranking to these candidate nodes.",
+                },
+            ),
+        },
+    },
+    returns => {
+        type => 'object',
+        properties => {
+            mode => $mode_return_property,
+            nodes => {
+                type => 'array',
+                description => "The candidate nodes, best first. The guest's current node and"
+                    . " nodes that are offline, in HA maintenance mode or without usage"
+                    . " information are omitted.",
+                items => {
+                    type => 'object',
+                    properties => {
+                        node => get_standard_option('pve-node'),
+                        imbalance => {
+                            type => 'number',
+                            optional => 1,
+                            description => "The cluster load imbalance, from 0 to 1, that"
+                                . " migrating the guest to this node would result in. Only"
+                                . " present for a running guest.",
+                        },
+                    },
+                },
+            },
+        },
+    },
+    code => sub($param) {
+        my $sid = guest_sid($param->{vmid}, 'vmid');
+
+        my $candidate_nodes =
+            $param->{'target-nodes'}
+            ? [PVE::Tools::split_list($param->{'target-nodes'})]
+            : undef;
+
+        my $planner = PVE::ResourceScheduling::Planner->build(
+            mode => $param->{mode} // 'auto',
+            nodes => schedulable_nodes(),
+            resolvers => static_stats_resolvers(),
+        );
+
+        return {
+            mode => $planner->mode(),
+            nodes => $planner->rank_target_nodes($sid, $candidate_nodes),
+        };
+    },
+});
+
+__PACKAGE__->register_method({
+    name => 'rebalance_plan',
+    path => 'rebalance-plan',
+    method => 'POST',
+    description =>
+        "Propose guest migrations that reduce the cluster load imbalance. Only running"
+        . " guests are considered, and locked guests are not proposed. Guests managed by HA are"
+        . " only proposed if HA would migrate them, that is, if their HA resource should be"
+        . " started and currently is, and no other resource has to move along. They are only"
+        . " proposed for nodes that HA considers online and their HA rules allow, taking the"
+        . " other proposed migrations into account. Guests that are not proposed still count"
+        . " towards the load of their node. Migrations that would barely improve the balance"
+        . " are not proposed.",
+    permissions => {
+        description => "The 'VM.Migrate' permission is required on '/vms/<ID>' for each ID"
+            . " passed via 'vms'. Without 'vms', only guests with 'VM.Migrate' are proposed, and"
+            . " the user needs that privilege on at least one guest.",
+        user => 'all',
+    },
+    parameters => {
+        additionalProperties => 0,
+        properties => {
+            vms => {
+                type => 'string',
+                format => 'pve-vmid-list',
+                optional => 1,
+                description =>
+                    "Only propose migrations of these guests (default: all guests the"
+                    . " user may migrate).",
+            },
+            'target-nodes' => get_standard_option(
+                'pve-node-list',
+                {
+                    optional => 1,
+                    description => "Only propose migrations to these nodes, for example to fill"
+                        . " newly added nodes. Guests can still be moved away from any node.",
+                },
+            ),
+            mode => $mode_property,
+            method => {
+                type => 'string',
+                enum => ['bruteforce', 'topsis'],
+                default => 'bruteforce',
+                optional => 1,
+                description => "The method used to score the candidate migrations.",
+            },
+            'max-moves' => {
+                type => 'integer',
+                minimum => 1,
+                maximum => 50,
+                default => 10,
+                optional => 1,
+                description => "The maximum number of migrations to propose.",
+            },
+            'min-improvement' => {
+                type => 'number',
+                minimum => 0,
+                maximum => 1,
+                default => 0,
+                optional => 1,
+                description =>
+                    "Only propose a migration if it lowers the imbalance by more than"
+                    . " this value, on the same 0 to 1 scale as the imbalance.",
+            },
+        },
+    },
+    returns => {
+        type => 'object',
+        properties => {
+            mode => $mode_return_property,
+            imbalance => {
+                type => 'object',
+                description => "The cluster load imbalance, from 0 (balanced) to 1.",
+                properties => {
+                    before => {
+                        type => 'number',
+                        description => "The current imbalance.",
+                    },
+                    after => {
+                        type => 'number',
+                        description => "The expected imbalance after all proposed migrations.",
+                    },
+                },
+            },
+            moves => {
+                type => 'array',
+                description => "The proposed migrations, at most one per guest.",
+                items => {
+                    type => 'object',
+                    properties => {
+                        vmid => get_standard_option('pve-vmid'),
+                        type => {
+                            type => 'string',
+                            enum => ['qemu', 'lxc'],
+                            description => "The guest type.",
+                        },
+                        source => get_standard_option(
+                            'pve-node',
+                            { description => "The node the guest currently runs on." },
+                        ),
+                        target => get_standard_option(
+                            'pve-node',
+                            { description => "The proposed target node." },
+                        ),
+                    },
+                },
+            },
+        },
+    },
+    code => sub($param) {
+        my $rpcenv = PVE::RPCEnvironment::get();
+        my $authuser = $rpcenv->get_user();
+
+        my $ids = PVE::Cluster::get_vmlist()->{ids} // {};
+
+        my @sids;
+        # an empty list is treated like an omitted one, so it cannot bypass the permission check
+        if (my @vmids = PVE::Tools::split_list($param->{vms} // '')) {
+            for my $vmid (@vmids) {
+                $rpcenv->check($authuser, "/vms/$vmid", ['VM.Migrate']);
+                push @sids, guest_sid($vmid, 'vms');
+            }
+        } else {
+            for my $vmid (sort keys %$ids) {
+                my $type = $guest_type_map->{ $ids->{$vmid}->{type} // '' } or next;
+                next if !$rpcenv->check($authuser, "/vms/$vmid", ['VM.Migrate'], 1);
+                push @sids, "$type:$vmid";
+            }
+            raise_perm_exc("VM.Migrate on at least one guest") if !@sids;
+        }
+
+        my $ha_status = PVE::HA::Config::read_manager_status();
+        my $nodes = schedulable_nodes($ha_status);
+
+        # a locked guest cannot be migrated right now, but still counts as load on its node
+        my $locks = PVE::Cluster::get_guest_config_properties(['lock']);
+        my $ha_states = PVE::HA::Config::get_resources_requested_state(\@sids, $ha_status);
+        my ($movable, $ha_sids) = ([], []);
+        for my $sid (@sids) {
+            my (undef, $vmid) = split(/:/, $sid, 2);
+            next if defined($locks->{$vmid}->{lock});
+            if (my $state = $ha_states->{$sid}) {
+                # HA stops a service configured to be stopped instead of migrating it, even if its
+                # current state is still started
+                push $ha_sids->@*, $sid if $state eq 'started';
+            } else {
+                push $movable->@*, $sid;
+            }
+        }
+
+        # HA only migrates a started service and none while HA is disarmed, only to a node it sees
+        # online, refuses a migration that its rules block and moves resources in positive affinity
+        # along. So propose an HA guest only if it is started and moves alone, and only to the
+        # online nodes its rules allow.
+        my $ha_nodes = $ha_status->{node_status} // {};
+        my $ha_services = $ha_status->{service_status} // {};
+        my $ha_started =
+            defined($ha_status->{disarm})
+            ? []
+            : [grep { ($ha_services->{$_}->{state} // '') eq 'started' } $ha_sids->@*];
+
+        my $snapshot_sids = {};
+        for my $vmid (keys %$ids) {
+            my $type = $guest_type_map->{ $ids->{$vmid}->{type} // '' } or next;
+            $snapshot_sids->{"$type:$vmid"} = 1;
+        }
+
+        my ($allowed_targets, $partners) = ({}, {});
+        if (scalar($ha_started->@*)) {
+            my $motion_info = PVE::HA::Config::get_resources_motion_info($ha_started);
+            for my $sid ($ha_started->@*) {
+                my $info = $motion_info->{$sid} // {};
+                next if scalar(($info->{'dependent-resources'} // [])->@*);
+
+                my $blocking = $info->{'blocking-resources-by-node'} // {};
+                $allowed_targets->{$sid} =
+                    [grep { ($ha_nodes->{$_} // '') eq 'online' && !$blocking->{$_} } $nodes->@*];
+                push $movable->@*, $sid;
+
+                # the blockers only reflect the current placement, so also keep guests in negative
+                # resource affinity apart from each other while the proposal moves them
+                for my $blocker (map { $_->@* } values %$blocking) {
+                    my $partner = $blocker->{sid};
+                    next if $blocker->{cause} ne 'resource-affinity';
+                    next if !$snapshot_sids->{$partner};
+                    $partners->{$sid}->{$partner} = 1;
+                    $partners->{$partner}->{$sid} = 1;
+                }
+            }
+        }
+        my $separate = { map { $_ => [sort keys $partners->{$_}->%*] } keys %$partners };
+
+        my $planner = PVE::ResourceScheduling::Planner->build(
+            mode => $param->{mode} // 'auto',
+            nodes => $nodes,
+            resolvers => static_stats_resolvers(),
+        );
+
+        my $plan = $planner->suggest_rebalance(
+            $movable,
+            method => $param->{method} // 'bruteforce',
+            max_moves => $param->{'max-moves'} // 10,
+            min_improvement => $param->{'min-improvement'} // 0,
+            targets => $param->{'target-nodes'}
+            ? [PVE::Tools::split_list($param->{'target-nodes'})]
+            : undef,
+            allowed_targets => $allowed_targets,
+            separate => $separate,
+        );
+
+        my $moves = [];
+        for my $move ($plan->{moves}->@*) {
+            my ($prefix, $vmid) = split(/:/, $move->{sid}, 2);
+            push $moves->@*,
+                {
+                    vmid => int($vmid),
+                    type => $guest_type_by_prefix->{$prefix},
+                    source => $move->{source},
+                    target => $move->{target},
+                };
+        }
+
+        return {
+            mode => $plan->{mode},
+            imbalance => $plan->{imbalance},
+            moves => $moves,
+        };
+    },
+});
+
+1;
diff --git a/test/Makefile b/test/Makefile
index 91761579d..5073c1bfc 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -5,7 +5,7 @@ all:
 export PERLLIB=..
 
 .PHONY: check
-check: test-replication test-balloon test-vzdump test-osd test-ceph-health test-ceph-services test-ceph-auth test-ceph-key-migration test-ceph-lockbox-migration test-custom-cpu-models test-pvesh test-report
+check: test-replication test-balloon test-vzdump test-osd test-ceph-health test-ceph-services test-ceph-auth test-ceph-key-migration test-ceph-lockbox-migration test-custom-cpu-models test-resource-scheduling test-pvesh test-report
 
 .PHONY: test-balloon
 test-balloon:
@@ -57,6 +57,10 @@ test-ceph-lockbox-migration:
 test-custom-cpu-models:
 	./CustomCPUModels_test.pl
 
+.PHONY: test-resource-scheduling
+test-resource-scheduling:
+	./ResourceScheduling_test.pl
+
 .PHONY: test-pvesh
 test-pvesh:
 	./pvesh_test.pl
diff --git a/test/ResourceScheduling_test.pl b/test/ResourceScheduling_test.pl
new file mode 100755
index 000000000..ecc1f8b25
--- /dev/null
+++ b/test/ResourceScheduling_test.pl
@@ -0,0 +1,454 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use lib ('.', '..');
+
+use JSON qw(encode_json);
+use Test::MockModule;
+use Test::More;
+
+use PVE::JSONSchema;
+use PVE::RPCEnvironment;
+
+use PVE::API2::Cluster::ResourceScheduling;
+
+my $GiB = 1024 * 1024 * 1024;
+
+# The cluster the mocked PVE::Cluster and HA getters below report, set up by new_cluster(). The
+# endpoints only read cluster state, so mocking these getters keeps the tests independent of a
+# running pmxcfs.
+my $cluster;
+
+# Per user, the ACL paths with 'VM.Migrate', the only privilege the endpoints check.
+my $migrate_acl = {};
+
+my $cluster_mock = Test::MockModule->new('PVE::Cluster');
+$cluster_mock->redefine(
+    get_members => sub() { return $cluster->{members} },
+    get_vmlist => sub() { return { ids => $cluster->{vmlist} } },
+    rrd_dump => sub() { return $cluster->{rrd} },
+    # callers decode the returned values in place, so hand out a copy
+    get_node_kv => sub($key, $node = undef) { return { ($cluster->{kv}->{$key} // {})->%* } },
+    get_guest_config_properties => sub($properties) {
+        my $res = {};
+        for my $vmid (keys $cluster->{confs}->%*) {
+            for my $prop (@$properties) {
+                my $value = $cluster->{confs}->{$vmid}->{$prop} // next;
+                $res->{$vmid}->{$prop} = $value;
+            }
+        }
+        return $res;
+    },
+);
+
+my $inotify_mock = Test::MockModule->new('PVE::INotify');
+$inotify_mock->redefine(nodename => sub() { return 'node1' });
+
+my $ha_config_mock = Test::MockModule->new('PVE::HA::Config');
+$ha_config_mock->redefine(read_manager_status => sub() { return $cluster->{ha_status} });
+# mock() instead of redefine(), as the batch helpers are newer than the pve-ha-manager a build may
+# have installed
+$ha_config_mock->mock(
+    # $cluster->{ha_managed} holds the configured HA state of each HA-managed guest
+    get_resources_requested_state => sub($sids, $manager_status = undef) {
+        my $res = {};
+        for my $sid (@$sids) {
+            my (undef, $vmid) = split(/:/, $sid, 2);
+            my $state = $cluster->{ha_managed}->{$vmid} // 'ignored';
+            $res->{$sid} = $state if $state ne 'ignored';
+        }
+        return $res;
+    },
+    get_resources_motion_info => sub($sids) {
+        push $cluster->{ha_motion_calls}->@*, [sort @$sids];
+        my $res = {};
+        for my $sid (@$sids) {
+            my (undef, $vmid) = split(/:/, $sid, 2);
+            next if !$cluster->{ha_managed}->{$vmid};
+            $res->{$sid} = $cluster->{ha_motion}->{$sid}
+                // { 'dependent-resources' => [], 'blocking-resources-by-node' => {} };
+        }
+        return $res;
+    },
+);
+
+my $rpcenv_mock = Test::MockModule->new('PVE::RPCEnvironment');
+$rpcenv_mock->redefine(
+    permissions => sub($self, $user, $path) {
+        return { 'VM.Migrate' => 1 }
+            if $user eq 'root@pam' || grep { $_ eq $path } ($migrate_acl->{$user} // [])->@*;
+        return {};
+    },
+);
+
+my $rpcenv = PVE::RPCEnvironment->init('cli');
+
+# Three online, idle nodes with 16 cores and 64 GiB of memory each, and no guests.
+sub new_cluster() {
+    $cluster = {
+        members => {},
+        vmlist => {},
+        rrd => {},
+        kv => { 'static-info' => {} },
+        confs => {},
+        ha_managed => {},
+        ha_motion => {},
+        ha_motion_calls => [],
+        ha_status => { node_status => {}, service_status => {} },
+    };
+
+    for my $node (qw(node1 node2 node3)) {
+        $cluster->{members}->{$node} = { online => 1 };
+        $cluster->{ha_status}->{node_status}->{$node} = 'online';
+        # RRD layout as broadcast by pvestatd: maxcpu, cpu, maxmem and mem at index 4, 5, 7 and 8
+        $cluster->{rrd}->{"pve-node-9.0/$node"} = [(undef) x 4, 16, 0, undef, 64 * $GiB, 0];
+        $cluster->{kv}->{'static-info'}->{$node} = encode_json({ cpus => 16, memory => 64 * $GiB });
+    }
+}
+
+# Add a guest with $cores cores and $mem_gib GiB of memory, by default a running VM on node1 that
+# uses 80% of its CPUs and 90% of its memory.
+sub add_guest($vmid, %opts) {
+    my $type = $opts{type} // 'qemu';
+    my $node = $opts{node} // 'node1';
+    my $running = $opts{running} // 1;
+    my ($cores, $mem) = ($opts{cores} // 2, ($opts{mem_gib} // 4) * $GiB);
+
+    $cluster->{vmlist}->{$vmid} = { type => $type, node => $node, version => 1 };
+
+    # RRD layout as broadcast by pvestatd: status, maxcpu, cpu, maxmem, mem at index 2, 5, 6, 7, 8
+    # and, for VMs, the host memory usage at index 15
+    my $used_mem = $running ? int(0.9 * $mem) : 0;
+    my $entry = [undef, undef, $running ? 'running' : 'stopped'];
+    $entry->@[5 .. 8] = ($cores, $running ? 0.8 : 0, $mem, $used_mem);
+    $entry->[15] = $used_mem if $type eq 'qemu';
+    $cluster->{rrd}->{"pve-vm-9.0/$vmid"} = $entry;
+
+    $cluster->{confs}->{$vmid} = { cores => $cores, memory => $mem / 1024 / 1024 };
+    $cluster->{confs}->{$vmid}->{lock} = $opts{lock} if defined($opts{lock});
+    if ($opts{ha}) {
+        $cluster->{ha_managed}->{$vmid} = 'started';
+        my $sid = ($type eq 'lxc' ? 'ct' : 'vm') . ":$vmid";
+        $cluster->{ha_status}->{service_status}->{$sid} = { state => 'started', node => $node };
+    }
+}
+
+# Let the HA rules of the HA-managed VM $vmid block migrations to the nodes in $opts{blocked}, keep
+# it apart from the VMs in $opts{apart} and move the resources in $opts{dependent} along with it.
+sub set_ha_motion($vmid, %opts) {
+    my $blocking = {};
+    for my $node (($opts{blocked} // [])->@*) {
+        push $blocking->{$node}->@*, { sid => "vm:$vmid", cause => 'node-affinity' };
+    }
+    # like HA, report a resource in negative affinity as blocking the node it currently runs on
+    for my $partner (($opts{apart} // [])->@*) {
+        my $node = $cluster->{vmlist}->{$partner}->{node};
+        push $blocking->{$node}->@*, { sid => "vm:$partner", cause => 'resource-affinity' };
+    }
+    $cluster->{ha_motion}->{"vm:$vmid"} = {
+        'dependent-resources' => $opts{dependent} // [],
+        'blocking-resources-by-node' => $blocking,
+    };
+}
+
+# Call an endpoint the way the API server does: validate the parameters, run the method's declared
+# permission check, then the handler, and check the result against the declared return schema.
+my $api_method = { 'recommend-target-nodes' => 'GET', 'rebalance-plan' => 'POST' };
+
+sub api_call($path, $user, $param = {}) {
+    my ($handler, $info) =
+        PVE::API2::Cluster::ResourceScheduling->find_handler($api_method->{$path}, $path, {});
+    die "no handler for '$path'\n" if !$info;
+
+    my $call_param = {%$param};
+    PVE::JSONSchema::validate($call_param, $info->{parameters});
+
+    $rpcenv->set_user($user);
+    $rpcenv->check_api2_permissions($info->{permissions}, $user, $call_param);
+
+    my $res = $handler->handle($info, $call_param);
+    PVE::JSONSchema::validate($res, $info->{returns}, "result does not match the return schema\n");
+
+    return $res;
+}
+
+# The exception an endpoint call raises, or undef if it succeeds.
+sub api_error($path, $user, $param = {}) {
+    eval { api_call($path, $user, $param) };
+    return $@ || undef;
+}
+
+# The ranked nodes, sorted by name: the tests check which nodes are candidates, not their order.
+sub candidate_nodes($ranking) {
+    return [sort map { $_->{node} } $ranking->{nodes}->@*];
+}
+
+sub moved_vmids($plan) {
+    return [sort { $a <=> $b } map { $_->{vmid} } $plan->{moves}->@*];
+}
+
+sub move_target($plan, $vmid) {
+    my ($move) = grep { $_->{vmid} == $vmid } $plan->{moves}->@*;
+    return $move ? $move->{target} : undef;
+}
+
+sub move_targets($plan) {
+    my $targets = { map { $_->{target} => 1 } $plan->{moves}->@* };
+    return [sort keys %$targets];
+}
+
+subtest 'recommend-target-nodes: permissions' => sub {
+    new_cluster();
+    add_guest($_) for 100, 101;
+    $migrate_acl = { 'op@pve' => ['/vms/101'] };
+
+    my $err = api_error('recommend-target-nodes', 'op@pve', { vmid => 100 });
+    is($err->{code}, 403, "guest without 'VM.Migrate' is rejected");
+
+    ok(api_call('recommend-target-nodes', 'op@pve', { vmid => 101 }),
+        "guest with 'VM.Migrate'");
+
+    $err = api_error('recommend-target-nodes', 'root@pam', { vmid => 999 });
+    is($err->{code}, 400, 'unknown guest is a parameter error');
+    like($err->{errors}->{vmid}, qr/does not exist/, 'error names the unknown guest');
+};
+
+subtest 'recommend-target-nodes: candidate nodes' => sub {
+    new_cluster();
+    add_guest(100);
+    add_guest(101, node => 'node2', running => 0);
+
+    my $ranking = api_call('recommend-target-nodes', 'root@pam', { vmid => 100 });
+    is($ranking->{mode}, 'dynamic', 'measured load is used if every node reports it');
+    is_deeply(candidate_nodes($ranking), [qw(node2 node3)], 'current node is excluded');
+    ok(!grep({ !defined($_->{imbalance}) } $ranking->{nodes}->@*), 'running guest: imbalance');
+
+    # a stopped guest is ranked as a placement, which has no resulting imbalance to report
+    $ranking = api_call('recommend-target-nodes', 'root@pam', { vmid => 101 });
+    is_deeply(candidate_nodes($ranking), [qw(node1 node3)], 'stopped guest: current excluded');
+    ok(!grep({ defined($_->{imbalance}) } $ranking->{nodes}->@*), 'stopped guest: none');
+
+    my $param = { vmid => 100, 'target-nodes' => 'node3' };
+    $ranking = api_call('recommend-target-nodes', 'root@pam', $param);
+    is_deeply(candidate_nodes($ranking), ['node3'], "ranking is limited to 'target-nodes'");
+
+    $cluster->{members}->{node2}->{online} = 0;
+    $ranking = api_call('recommend-target-nodes', 'root@pam', { vmid => 100 });
+    is_deeply(candidate_nodes($ranking), ['node3'], 'offline node is excluded');
+
+    # a node in maintenance is about to be drained, yet it would rank first as the least loaded one
+    $cluster->{members}->{node2}->{online} = 1;
+    $cluster->{ha_status}->{node_status}->{node3} = 'maintenance';
+    $ranking = api_call('recommend-target-nodes', 'root@pam', { vmid => 100 });
+    is_deeply(candidate_nodes($ranking), ['node2'], 'node in HA maintenance mode is excluded');
+
+    # a node without any usage information must not fail the request, it is left out instead
+    $cluster->{ha_status}->{node_status} = {};
+    delete $cluster->{rrd}->{'pve-node-9.0/node3'};
+    delete $cluster->{kv}->{'static-info'}->{node3};
+    $ranking = api_call('recommend-target-nodes', 'root@pam', { vmid => 100 });
+    is($ranking->{mode}, 'static', 'missing load of a node falls back to static usage');
+    is_deeply(candidate_nodes($ranking), ['node2'], 'node without usage data is excluded');
+};
+
+subtest 'rebalance-plan: permissions' => sub {
+    new_cluster();
+    add_guest($_) for 100 .. 103;
+    $migrate_acl = { 'op@pve' => ['/vms/101'] };
+
+    # an empty or separator-only list must behave like an omitted one, so it cannot bypass the check
+    my $no_vms = { omitted => {}, empty => { vms => '' }, 'separator-only' => { vms => ',;' } };
+
+    # without any migratable guest the user could only learn the cluster load, so refuse
+    for my $case (sort keys %$no_vms) {
+        my $err = api_error('rebalance-plan', 'nobody@pve', $no_vms->{$case});
+        is($err->{code}, 403, "no privileges, 'vms' $case: rejected");
+    }
+
+    for my $case (sort keys %$no_vms) {
+        my $plan = api_call('rebalance-plan', 'op@pve', $no_vms->{$case});
+        is_deeply(moved_vmids($plan), [101], "'vms' $case: only the permitted guest moves");
+    }
+
+    my $err = api_error('rebalance-plan', 'op@pve', { vms => '101,102' });
+    is($err->{code}, 403, "explicit guest without 'VM.Migrate' is rejected");
+
+    # the permission check comes first, so the error does not reveal whether a guest exists
+    $err = api_error('rebalance-plan', 'op@pve', { vms => '101,999' });
+    is($err->{code}, 403, 'unknown guest without privileges is a permission error');
+
+    $err = api_error('rebalance-plan', 'root@pam', { vms => '999' });
+    is($err->{code}, 400, 'unknown guest is a parameter error');
+    like($err->{errors}->{vms}, qr/does not exist/, 'error names the unknown guest');
+};
+
+subtest 'rebalance-plan: locked guests stay put' => sub {
+    # the two big guests are the most effective ones to move off the overloaded node1
+    new_cluster();
+    add_guest(100, cores => 8, mem_gib => 16);
+    add_guest(101, cores => 8, mem_gib => 16);
+    add_guest(102);
+    add_guest(103, type => 'lxc');
+
+    my $unlocked = api_call('rebalance-plan', 'root@pam');
+    ok(defined(move_target($unlocked, 101)), 'without a lock, the big guest is proposed');
+
+    add_guest(101, cores => 8, mem_gib => 16, lock => 'backup');
+
+    my $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 101)), 'locked guest is not moved');
+    is(
+        $plan->{imbalance}->{before},
+        $unlocked->{imbalance}->{before},
+        'locked guest still counts as load on its node',
+    );
+};
+
+subtest 'rebalance-plan: HA-managed guests follow their HA rules' => sub {
+    # as above, the big guests 100 and 101 are the most effective ones to move
+    new_cluster();
+    add_guest(100, cores => 8, mem_gib => 16, ha => 1);
+    add_guest(101, cores => 8, mem_gib => 16);
+    add_guest(102);
+    add_guest(103, type => 'lxc');
+
+    my $free = api_call('rebalance-plan', 'root@pam');
+    ok(defined(move_target($free, 100)), 'HA-managed guest is proposed');
+    my $calls = $cluster->{ha_motion_calls};
+    is_deeply($calls, [['vm:100']], 'HA rules are queried once, only for HA-managed guests');
+
+    set_ha_motion(100, blocked => ['node2']);
+    my $plan = api_call('rebalance-plan', 'root@pam');
+    is(move_target($plan, 100), 'node3', 'HA-managed guest only goes to an allowed node');
+    is(move_target($plan, 101), 'node2', 'node blocked for HA is still a target for others');
+
+    set_ha_motion(100, blocked => ['node2', 'node3']);
+    $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 100)), 'HA-managed guest without an allowed node stays');
+    ok(defined(move_target($plan, 101)), 'other guests are still proposed');
+    is(
+        $plan->{imbalance}->{before},
+        $free->{imbalance}->{before},
+        'HA-managed guest that stays still counts as load on its node',
+    );
+
+    # HA would migrate the dependent resources along, which a per-guest proposal cannot express
+    set_ha_motion(100, dependent => ['vm:104']);
+    $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 100)), 'HA-managed guest with dependent resources stays');
+
+    # HA drops a migration to a node it does not see online, even if the cluster does
+    set_ha_motion(100, blocked => ['node2']);
+    $cluster->{ha_status}->{node_status}->{node3} = 'unknown';
+    $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 100)),
+        'HA-managed guest only goes to a node HA sees online');
+};
+
+subtest 'rebalance-plan: HA-managed guests only if HA would migrate them' => sub {
+    # as above, the big guests 100 and 101 are the most effective ones to move
+    new_cluster();
+    add_guest(100, cores => 8, mem_gib => 16, ha => 1);
+    add_guest(101, cores => 8, mem_gib => 16);
+    add_guest(102);
+    add_guest(103, type => 'lxc');
+
+    my $started = api_call('rebalance-plan', 'root@pam');
+    ok(defined(move_target($started, 100)), 'started HA service is proposed');
+    my $load = $started->{imbalance}->{before};
+
+    # until HA processes a changed configuration, the current state can still read started, but HA
+    # then stops the guest instead of migrating it
+    for my $configured (qw(stopped disabled)) {
+        $cluster->{ha_managed}->{100} = $configured;
+        my $plan = api_call('rebalance-plan', 'root@pam');
+        ok(!defined(move_target($plan, 100)), "HA service configured as $configured stays");
+        is($plan->{imbalance}->{before}, $load, 'and still counts as load');
+    }
+    $cluster->{ha_managed}->{100} = 'started';
+
+    # a service that is migrating, recovering or the like is busy with HA's own plans
+    $cluster->{ha_status}->{service_status}->{'vm:100'}->{state} = 'migrate';
+    my $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 100)), 'HA service in another state stays');
+    is($plan->{imbalance}->{before}, $load, 'and still counts as load');
+
+    # a frozen HA stack still owns its guests, but does not carry out any migration request
+    $cluster->{ha_status}->{service_status}->{'vm:100'}->{state} = 'freeze';
+    $cluster->{ha_status}->{disarm} = { mode => 'freeze', state => 'disarmed' };
+    $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 100)), 'HA-managed guest stays while HA is frozen');
+    ok(defined(move_target($plan, 101)), 'other guests are still proposed');
+    is($plan->{imbalance}->{before}, $load, 'and still counts as load');
+
+    # while disarming, the services may still be marked started for a moment
+    $cluster->{ha_status}->{service_status}->{'vm:100'}->{state} = 'started';
+    $cluster->{ha_status}->{disarm} = { mode => 'freeze', state => 'disarming' };
+    $plan = api_call('rebalance-plan', 'root@pam');
+    ok(!defined(move_target($plan, 100)), 'HA-managed guest stays while HA is disarming');
+};
+
+subtest 'rebalance-plan: negative resource affinity holds across the proposal' => sub {
+    # node1 and node2 are busy and node3 is idle, so it is the best target for 101 and 102 alike,
+    # which their negative resource affinity forbids
+    new_cluster();
+    $cluster->{rrd}->{"pve-node-9.0/$_"}->@[5, 8] = (0.875, 56 * $GiB) for qw(node1 node2);
+    add_guest(101, cores => 4, mem_gib => 16, ha => 1);
+    add_guest(102, cores => 4, mem_gib => 16, ha => 1, node => 'node2');
+    set_ha_motion(101, apart => [102]);
+    set_ha_motion(102, apart => [101]);
+
+    my $plan = api_call('rebalance-plan', 'root@pam');
+    my @targets = grep { defined } map { move_target($plan, $_) } 101, 102;
+    is_deeply(\@targets, ['node3'], 'only one of the separated guests moves to the idle node');
+
+    # knowing the affinity from one side suffices to keep the two apart
+    set_ha_motion(102);
+    $plan = api_call('rebalance-plan', 'root@pam');
+    @targets = grep { defined } map { move_target($plan, $_) } 101, 102;
+    is_deeply(\@targets, ['node3'], 'one-sided affinity also keeps them apart');
+};
+
+subtest 'rebalance-plan: proposal' => sub {
+    # the big container is the most effective guest to move, so the proposal includes a container
+    new_cluster();
+    add_guest($_) for 100 .. 102;
+    add_guest(103, type => 'lxc', cores => 8, mem_gib => 16);
+    add_guest(104, running => 0);
+
+    my $plan = api_call('rebalance-plan', 'root@pam');
+    ok(scalar($plan->{moves}->@*), 'overloaded node gets a proposal');
+    my $imbalance = $plan->{imbalance};
+    ok($imbalance->{after} < $imbalance->{before}, 'proposal lowers the imbalance');
+    ok(!grep({ $_ == 104 } moved_vmids($plan)->@*), 'stopped guest is not moved');
+
+    # the result carries plain guest ids and types, not the scheduler's internal resource ids
+    for my $move ($plan->{moves}->@*) {
+        my ($vmid, $guest) = ($move->{vmid}, $cluster->{vmlist}->{ $move->{vmid} });
+        is_deeply([sort keys %$move], [qw(source target type vmid)], "move of $vmid: keys");
+        is($move->{type}, $guest->{type}, "move of $vmid: type");
+        is($move->{source}, $guest->{node}, "move of $vmid: source");
+    }
+    ok((grep { $_->{type} eq 'lxc' } $plan->{moves}->@*), 'container is reported as lxc');
+
+    $plan = api_call('rebalance-plan', 'root@pam', { 'max-moves' => 1 });
+    is(scalar($plan->{moves}->@*), 1, "'max-moves' caps the proposal");
+
+    for my $max_moves (0, 51) {
+        my $err = api_error('rebalance-plan', 'root@pam', { 'max-moves' => $max_moves });
+        ok($err->{errors}->{'max-moves'}, "'max-moves' $max_moves is out of range");
+    }
+
+    # restricting the targets, e.g. to fill a new node, still moves guests off any other node
+    $plan = api_call('rebalance-plan', 'root@pam', { 'target-nodes' => 'node2' });
+    ok(scalar($plan->{moves}->@*), "proposal with 'target-nodes'");
+    is_deeply(move_targets($plan), ['node2'], "only 'target-nodes' are targets");
+
+    $cluster->{ha_status}->{node_status}->{node2} = 'maintenance';
+    $plan = api_call('rebalance-plan', 'root@pam');
+    ok(scalar($plan->{moves}->@*), 'proposal with a node in HA maintenance mode');
+    is_deeply(move_targets($plan), ['node3'], 'HA maintenance node is no target');
+};
+
+done_testing();
-- 
2.47.3





  parent reply	other threads:[~2026-11-18 11:09 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 11:05 [PATCH guest-common,ha-manager,manager,docs 0/15] cluster resource scheduling for all guests Thomas Lamprecht
2026-11-18 11:05 ` [PATCH guest-common 01/15] scheduling: add cluster resource scheduling library Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 02/15] helpers: block both locations of a migrating negative affinity partner Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 03/15] usage: static: count services per node again for scoring fallback Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 04/15] config: allow getting motion info of many resources at once Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 05/15] env: pve2: delegate cluster stat gathering to scheduling library Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 06/15] usage: share one scheduler base between static and dynamic Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 07/15] ui: bulk action: translate error title and CT restart warning Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 08/15] ui: bulk migrate: show CT restart warning for cluster-wide migration Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 09/15] ui: migrate: recheck preconditions if the target changed meanwhile Thomas Lamprecht
2026-11-18 11:05 ` Thomas Lamprecht [this message]
2026-11-18 11:05 ` [PATCH manager 11/15] ui: migrate: sort target nodes by scheduler recommendation Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 12/15] api: bulk migrate: allow a per-guest target node mapping Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 13/15] ui: bulk migrate: support per-guest target nodes Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 14/15] ui: dc: add a Rebalance Guests bulk action Thomas Lamprecht
2026-11-18 11:05 ` [PATCH docs 15/15] pvecm: document balancing the cluster load with migrations Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118110705.595989-11-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal