From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH ha-manager 04/15] config: allow getting motion info of many resources at once
Date: Wed, 18 Nov 2026 12:05:11 +0100 [thread overview]
Message-ID: <20261118110705.595989-5-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118110705.595989-1-t.lamprecht@proxmox.com>
The cluster rebalance proposal of the resource scheduling API needs
to know, for possibly every guest in the cluster, which nodes its HA
rules allow and which HA resources move together with it. Getting
that through get_resource_motion_info reads and compiles the whole
rule set again for each resource, which scales poorly with the number
of resources in a request.
Add get_resources_motion_info, which reads the configs and compiles
the rules once and returns the motion info for each HA-managed
resource of the given list. Turn get_resource_motion_info into a thin
wrapper around it, so both share one implementation and its existing
callers, such as the migrate preconditions of qemu-server and
pve-container, see no change.
Each read of the manager status returns a fresh copy, which gets
costly for many resources in a large cluster. So pass the manager
status that was already read to the HA state check instead of reading
it again for every resource.
For the same reason, add get_resources_requested_state, which tells
the HA-managed resources of a list and their requested state with one
read of the resources config, while vm_is_ha_managed copies the whole
config for each guest.
Extend the motion info test to cover the new helpers.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/HA/Config.pm | 83 ++++++++++++++++++++-------
src/test/test_resource_motion_info.pl | 43 ++++++++++++++
2 files changed, 106 insertions(+), 20 deletions(-)
diff --git a/src/PVE/HA/Config.pm b/src/PVE/HA/Config.pm
index bee1ac7..4e4e8b2 100644
--- a/src/PVE/HA/Config.pm
+++ b/src/PVE/HA/Config.pm
@@ -353,14 +353,14 @@ sub read_crm_commands {
}
my $service_check_ha_state = sub {
- my ($conf, $sid, $has_state) = @_;
+ my ($conf, $sid, $has_state, $manager_status) = @_;
if (my $d = $conf->{ids}->{$sid}) {
if (!defined($has_state)) {
# ignored service behave as if they were not managed by HA
return 0 if defined($d->{state}) && $d->{state} eq 'ignored';
# cluster-wide disarm with ignore mode - resources can be managed directly
- my $ms = cfs_read_file($manager_status_filename);
+ my $ms = $manager_status // cfs_read_file($manager_status_filename);
if (my $disarm = $ms->{disarm}) {
return 0 if $disarm->{mode} eq 'ignore';
}
@@ -391,36 +391,79 @@ sub service_is_configured {
return 0;
}
-sub get_resource_motion_info {
- my ($sid) = @_;
+# Returns the motion info of each HA-managed resource in $sids as a hash of
+# $sid => { 'dependent-resources' => [...], 'blocking-resources-by-node' => { $node => [...] } }
+# as described for PVE::HA::Helpers::get_resource_motion_info(). Resources that are not
+# HA-managed are omitted. The rules are read and compiled once, independent of the number of
+# resources.
+sub get_resources_motion_info {
+ my ($sids) = @_;
my $cfg = read_resources_config();
+ # read once, as each copy of a large manager status is costly
+ my $manager_status = read_manager_status();
- my $dependent_resources = [];
- my $blocking_resources_by_node = {};
+ my $managed_sids =
+ [grep { $service_check_ha_state->($cfg, $_, undef, $manager_status) } @$sids];
+ return {} if !@$managed_sids;
- if (&$service_check_ha_state($cfg, $sid)) {
- my $manager_status = read_manager_status();
- my $ss = $manager_status->{service_status};
- my $ns = $manager_status->{node_status};
- # get_resource_motion_info expects a hashset of all nodes with status 'online'
- my $online_nodes = { map { $ns->{$_} eq 'online' ? ($_ => 1) : () } keys %$ns };
- # get_resource_motion_info expects a resource config with defaults set
- my $resources = checked_resources_config($cfg);
+ my $ss = $manager_status->{service_status};
+ my $ns = $manager_status->{node_status};
+ # get_resource_motion_info expects a hashset of all nodes with status 'online'
+ my $online_nodes = { map { $ns->{$_} eq 'online' ? ($_ => 1) : () } keys %$ns };
+ # get_resource_motion_info expects a resource config with defaults set
+ my $resources = checked_resources_config($cfg);
- my $rules = read_and_check_rules_config();
- # TODO PVE 10: Remove group migration when HA groups have been fully migrated to rules
- my $groups = read_group_config();
- my $compiled_rules = compile_rules_config($rules, $groups, $resources, $manager_status);
+ my $rules = read_and_check_rules_config();
+ # TODO PVE 10: Remove group migration when HA groups have been fully migrated to rules
+ my $groups = read_group_config();
+ my $compiled_rules = compile_rules_config($rules, $groups, $resources, $manager_status);
+ my $motion_info = {};
+ for my $sid (@$managed_sids) {
my $cd = $resources->{$sid} // {};
- ($dependent_resources, $blocking_resources_by_node) =
+ my ($dependent_resources, $blocking_resources_by_node) =
PVE::HA::Helpers::get_resource_motion_info(
$ss, $sid, $cd, $online_nodes, $compiled_rules,
);
+
+ $motion_info->{$sid} = {
+ 'dependent-resources' => $dependent_resources,
+ 'blocking-resources-by-node' => $blocking_resources_by_node,
+ };
}
- return ($dependent_resources, $blocking_resources_by_node);
+ return $motion_info;
+}
+
+# Returns the requested state of each HA-managed resource in $sids as a hash of $sid => $state,
+# reading the resources config only once. Resources that are not HA-managed are omitted, as for
+# service_is_ha_managed(). Pass the manager status if it was already read.
+sub get_resources_requested_state {
+ my ($sids, $manager_status) = @_;
+
+ my $cfg = read_resources_config();
+ $manager_status //= read_manager_status();
+
+ my $states = {};
+ for my $sid (@$sids) {
+ next if !$service_check_ha_state->($cfg, $sid, undef, $manager_status);
+ my $state = $cfg->{ids}->{$sid}->{state} // 'started';
+ $states->{$sid} = $state eq 'enabled' ? 'started' : $state;
+ }
+
+ return $states;
+}
+
+sub get_resource_motion_info {
+ my ($sid) = @_;
+
+ my $motion_info = get_resources_motion_info([$sid])->{$sid} // {};
+
+ return (
+ $motion_info->{'dependent-resources'} // [],
+ $motion_info->{'blocking-resources-by-node'} // {},
+ );
}
# graceful, as long as locking + cfs_write works
diff --git a/src/test/test_resource_motion_info.pl b/src/test/test_resource_motion_info.pl
index 2f7c3cd..9729ca6 100755
--- a/src/test/test_resource_motion_info.pl
+++ b/src/test/test_resource_motion_info.pl
@@ -148,6 +148,30 @@ for my $sid (qw(vm:103 vm:999)) {
);
}
+my $compile_count = 0;
+my $rules_module = Test::MockModule->new('PVE::HA::Rules');
+$rules_module->redefine(
+ compile => sub {
+ $compile_count++;
+ return $rules_module->original('compile')->(@_);
+ },
+);
+
+is_deeply(
+ PVE::HA::Config::get_resources_motion_info([sort(keys %$expected), 'vm:103', 'vm:999']),
+ $expected,
+ "motion info for multiple resources omits ignored and unmanaged ones",
+);
+is($compile_count, 1, "rules are compiled once for multiple resources");
+
+is_deeply(PVE::HA::Config::get_resources_motion_info([]), {}, "no motion info for no resources");
+
+is_deeply(
+ PVE::HA::Config::get_resources_requested_state([qw(vm:101 vm:103 ct:104 vm:999)]),
+ { 'vm:101' => 'started', 'ct:104' => 'started' },
+ "requested state of the HA-managed resources, without ignored and unmanaged ones",
+);
+
# a migrating negative affinity partner occupies both its source and its target node
$manager_status->{service_status}->{'vm:102'} =
{ state => 'migrate', node => 'node3', target => 'node2' };
@@ -182,4 +206,23 @@ my $expected_migrating = {
$test_single_motion_info->($expected_migrating, 'migrating partner');
+is_deeply(
+ PVE::HA::Config::get_resources_motion_info([sort keys %$expected_migrating]),
+ $expected_migrating,
+ "migrating partner: motion info for multiple resources",
+);
+
+$manager_status->{disarm} = { mode => 'ignore' };
+
+is_deeply(
+ PVE::HA::Config::get_resources_motion_info([sort keys %$expected]),
+ {},
+ "no motion info while HA is disarmed with ignore mode",
+);
+is_deeply(
+ [PVE::HA::Config::get_resource_motion_info('vm:101')],
+ [[], {}],
+ "no single motion info while HA is disarmed with ignore mode",
+);
+
done_testing();
--
2.47.3
next prev parent reply other threads:[~2026-11-18 11:07 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 11:05 [PATCH guest-common,ha-manager,manager,docs 0/15] cluster resource scheduling for all guests Thomas Lamprecht
2026-11-18 11:05 ` [PATCH guest-common 01/15] scheduling: add cluster resource scheduling library Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 02/15] helpers: block both locations of a migrating negative affinity partner Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 03/15] usage: static: count services per node again for scoring fallback Thomas Lamprecht
2026-11-18 11:05 ` Thomas Lamprecht [this message]
2026-11-18 11:05 ` [PATCH ha-manager 05/15] env: pve2: delegate cluster stat gathering to scheduling library Thomas Lamprecht
2026-11-18 11:05 ` [PATCH ha-manager 06/15] usage: share one scheduler base between static and dynamic Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 07/15] ui: bulk action: translate error title and CT restart warning Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 08/15] ui: bulk migrate: show CT restart warning for cluster-wide migration Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 09/15] ui: migrate: recheck preconditions if the target changed meanwhile Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 10/15] api: cluster: add resource scheduling endpoints Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 11/15] ui: migrate: sort target nodes by scheduler recommendation Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 12/15] api: bulk migrate: allow a per-guest target node mapping Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 13/15] ui: bulk migrate: support per-guest target nodes Thomas Lamprecht
2026-11-18 11:05 ` [PATCH manager 14/15] ui: dc: add a Rebalance Guests bulk action Thomas Lamprecht
2026-11-18 11:05 ` [PATCH docs 15/15] pvecm: document balancing the cluster load with migrations Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118110705.595989-5-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox