[RFC qemu-server 0/4] remote migrate: extract preconditions and add check endpoint

[RFC qemu-server 0/4] remote migrate: extract preconditions and add check endpoint

[Erik Fastermann]

This series reworks how the QEMU remote-migration precondition checks
are structured and exposes them through a new endpoint, so blockers and
warnings can be surfaced before a migration is started rather than
mid-flight.

It is deliberately sent as an RFC: the change touches a critical path,
and the larger direction (see the open questions below) matters more
than the current diff. Thanks to @Fabian Gruenbichler for the discussion
so far.

Motivation
==========

Today the checks run at the very start of a remote migration and abort
on the first error via die. Two problems follow:

- Many prerequisites are not checked up front at all; they only surface
  once the migration is already running, e.g. local/mapped devices or a
  VNC clipboard that is not live-migratable. A user fixes one blocker,
  retries, and hits the next. A forum user collected a checklist of such
  prerequisites [0] (thanks @Arthur Bied-Charreton for pointing this
  out).

- The qm CLI wrapper only ran a subset of the checks the API path ran,
  so direct API callers (e.g. the web UI) and the CLI disagreed on what
  was validated.

What the series does
====================

1. Drop the ineffective fingerprint auto-detection.

2. Extract the checks into a validate_remote_migrate_preconditions
   helper.

3. Register the remote-migrate command against PVE::API2::Qemu
   directly and drop the CLI wrapper, so CLI and API run the exact same
   checks.

4. Add a remote_migrate_vm_precondition endpoint that runs the checks
   without starting a migration and returns the full findings list. It
   reuses the same helper as the migrate endpoint, so the precheck
   cannot drift from what is actually enforced.

The precondition results are returned as a flat list of findings. Does
this shape look reasonable, and could a similar structure be reused for
the intra-cluster migration precondition endpoint?

Open questions (RFC)
====================

Scope of the checks
- The checks should stay semi-static, i.e. mostly config-derived, and
  run quickly, i.e. avoid slow probes such as querying storage
  contents. But we still want to query the remote, since a precheck
  that never talks to the target is not very useful. Where/how can we
  reasonably reduce the number of API round trips to the remote?

Where should the checks run?
- The source node currently runs the checks with a few local and fast
  remote API calls. Since we want migrations between major versions to
  work in a mostly forward- and backward-compatible way (older source
  to newer target being the typical case), it might be cleaner to run
  the checks on the remote by sending the relevant local config over.
  When the remote is newer, it can also run more or patched checks.
  Of course this would be a larger change, as the general structure
  currently goes from source to target.
- Collecting results from both sides in PDM and comparing there seems
  wrong: the checks still have to run locally for the real migration,
  and both paths would then need to agree. We should rather keep a
  single authority.
- Running the checks over the Tunnel command instead would add a new
  command that older endpoints do not understand. Both here and for the
  API, a capability/feature endpoint could help: source and/or target
  query what each supports for a migration, and older versions fall
  back to the current checks. Is that worth introducing now?

Two-phase refactor
- The cleaner fix for "checks only run mid-migration" is to extract all
  preconditions and split execution into two phases. Phase 1 is shared
  by the precondition endpoint and the migration: it runs all the
  reasonable checks and builds a plan hash of the queried info. Phase 2
  performs the actual migration, ideally with its remaining checks up
  front and ordered fast-to-slow. This is a large refactor of a
  critical path, as the cluster migration code is mostly shared with
  the remote migration. In addition, the cluster migration precondition
  endpoint can also be refactored, which is an open TODO. Is that the
  direction we want?

Containers
- A similar series could later cover remote container migration. What
  pitfalls should I watch out for there?

References
==========
[0] https://forum.proxmox.com/threads/pdm-cross-cluster-migration-prerequisites-checklist-i-wish-i-had-before-my-first-attempt.184485

Thanks in advance for the feedback!


Erik Fastermann (4):
  remote migrate: drop ineffective fingerprint auto-detection
  remote migrate: collect preconditions as structured findings
  qm: remote-migrate: call API endpoint directly
  remote migrate: add precondition check endpoint

 src/PVE/API2/Qemu.pm | 472 +++++++++++++++++++++++++++++--------------
 src/PVE/CLI/qm.pm    | 128 +-----------
 2 files changed, 323 insertions(+), 277 deletions(-)

-- 
2.47.3

[RFC qemu-server 1/4] remote migrate: drop ineffective fingerprint auto-detection

[Erik Fastermann]

When no fingerprint was supplied, the code fetched the remote node
certificate and pinned its fingerprint for the migration tunnel. This
has no functional effect and is removed.

That fetch only succeeds for remotes trusted via the CA store, which
is exactly the case where pinning is unnecessary: both the API client
and the websocket tunnel fall back to CA verification on their own.
For a self-signed remote the fetch itself fails verification, so no
fingerprint is ever obtained.

The only meaningful path, an explicitly supplied fingerprint (needed to
verify self-signed remotes), is unchanged.

Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
---
 src/PVE/API2/Qemu.pm | 16 ++--------------
 1 file changed, 2 insertions(+), 14 deletions(-)

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 28cbb9b0..68f1800c 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -5739,26 +5739,14 @@ __PACKAGE__->register_method({
             apitoken => $remote->{apitoken},
         };
 
-        my $fp;
-        if ($fp = $remote->{fingerprint}) {
-            $conn_args->{cached_fingerprints} = { uc($fp) => 1 };
+        if ($remote->{fingerprint}) {
+            $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
         }
 
         print "Establishing API connection with remote at '$remote->{host}'\n";
 
         my $api_client = PVE::APIClient::LWP->new(%$conn_args);
 
-        if (!defined($fp)) {
-            my $cert_info = $api_client->get("/nodes/localhost/certificates/info");
-            foreach my $cert (@$cert_info) {
-                my $filename = $cert->{filename};
-                next if $filename ne 'pveproxy-ssl.pem' && $filename ne 'pve-ssl.pem';
-                $fp = $cert->{fingerprint} if !$fp || $filename eq 'pveproxy-ssl.pem';
-            }
-            $conn_args->{cached_fingerprints} = { uc($fp) => 1 }
-                if defined($fp);
-        }
-
         my $repl_conf = PVE::ReplicationConfig->new();
         my $is_replicated = $repl_conf->check_for_existing_jobs($source_vmid, 1);
         die "cannot remote-migrate replicated VM\n" if $is_replicated;
-- 
2.47.3

[RFC qemu-server 2/4] remote migrate: collect preconditions as structured findings

[Erik Fastermann]

Extract the remote-migration precondition checks into a
validate_remote_migrate_preconditions helper that records each problem
as a {severity, code, message} finding and returns the derived
migration parameters. The endpoint fails on the first error finding,
preserving the previous abort behavior.

It also pulls the checks that previously lived only in the qm CLI into
the endpoint. Direct API callers such as the web UI now run them too,
so fewer migrations start only to fail partway. This prepares for a
precondition endpoint that returns the full findings list instead of
dying on the first error.

User-visible effect is minimal: successful migrations are unchanged.
Some precondition error messages are reworded and the "Establishing
API connection" info line is no longer printed.

Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
---
 src/PVE/API2/Qemu.pm | 304 ++++++++++++++++++++++++++++++-------------
 1 file changed, 211 insertions(+), 93 deletions(-)

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 68f1800c..9b487457 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -1042,6 +1042,195 @@ sub assert_scsi_feature_compatibility {
     }
 }
 
+my sub validate_remote_migrate_preconditions {
+    my ($param, $findings) = @_;
+
+    my $add_error = sub {
+        push @$findings,
+            {
+                severity => 'error',
+                code => $_[0],
+                message => ("$_[1]" =~ s/\s+$//r),
+                @_[2 .. $#_],
+            };
+    };
+
+    my $add_warning = sub {
+        push @$findings,
+            {
+                severity => 'warning',
+                code => $_[0],
+                message => ("$_[1]" =~ s/\s+$//r),
+                @_[2 .. $#_],
+            };
+    };
+
+    my $source_vmid = extract_param($param, 'vmid');
+    my $target_endpoint = extract_param($param, 'target-endpoint');
+    my $remote = PVE::JSONSchema::parse_property_string('proxmox-remote', $target_endpoint);
+    my $target_vmid = extract_param($param, 'target-vmid') // $source_vmid;
+
+    my $target_storage = extract_param($param, 'target-storage');
+    my $storagemap = eval { PVE::JSONSchema::parse_idmap($target_storage, 'pve-storage-id') };
+    raise_param_exc({ 'target-storage' => "failed to parse storage map: $@" })
+        if $@;
+
+    my $target_bridge = extract_param($param, 'target-bridge');
+    my $bridgemap = eval { PVE::JSONSchema::parse_idmap($target_bridge, 'pve-bridge-id') };
+    raise_param_exc({ 'target-bridge' => "failed to parse bridge map: $@" })
+        if $@;
+
+    my $delete = extract_param($param, 'delete') // 0;
+
+    PVE::Cluster::check_cfs_quorum();
+
+    # test if VM exists
+    my $conf = eval { PVE::QemuConfig->load_config($source_vmid) };
+    if ($@) {
+        $add_error->('load-vm-config', $@);
+        return;
+    }
+
+    if (PVE::QemuConfig->has_lock($conf)) {
+        $add_error->('vm-locked', "VM is locked ($conf->{lock})");
+    }
+
+    if (PVE::HA::Config::service_is_configured("vm:$source_vmid")) {
+        $add_error->('vm-ha-configured', 'cannot remote-migrate VM that is configured for HA');
+    }
+
+    my $conn_args = {
+        protocol => 'https',
+        host => $remote->{host},
+        port => $remote->{port} // 8006,
+        apitoken => $remote->{apitoken},
+    };
+
+    if ($remote->{fingerprint}) {
+        $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
+    }
+
+    my $api_client = PVE::APIClient::LWP->new(%$conn_args);
+
+    # check connection once at the start
+    eval { $api_client->get("/version") };
+    if ($@) {
+        $add_error->('remote-conn', $@);
+        return;
+    }
+
+    my $resources = $api_client->get("/cluster/resources", { type => 'vm' });
+    if (grep { defined($_->{vmid}) && $_->{vmid} eq $target_vmid } @$resources) {
+        $add_error->('target-vmid-exists', "remote: guest with ID '$target_vmid' already exists");
+    }
+
+    my $storages = $api_client->get("/nodes/localhost/storage", { enabled => 1 });
+
+    my $check_remote_storage = sub {
+        my ($storage) = @_;
+        my $found = [grep { $_->{storage} eq $storage } @$storages];
+
+        if (!@$found) {
+            $add_error->(
+                'storage-missing',
+                "remote: storage '$storage' does not exist (or missing permission)",
+                storage => $storage,
+            );
+            return;
+        }
+
+        $found = $found->[0];
+        my $content_types = [PVE::Tools::split_list($found->{content})];
+        $add_error->(
+            'storage-no-images',
+            "remote: storage '$storage' cannot store images",
+            storage => $storage,
+        ) if !grep { $_ eq 'images' } @$content_types;
+    };
+
+    foreach my $target_sid (values %{ $storagemap->{entries} }) {
+        $check_remote_storage->($target_sid);
+    }
+
+    $check_remote_storage->($storagemap->{default})
+        if $storagemap->{default};
+
+    my $repl_conf = PVE::ReplicationConfig->new();
+    my $is_replicated = $repl_conf->check_for_existing_jobs($source_vmid, 1);
+    $add_error->('vm-replicated', "cannot remote-migrate replicated VM")
+        if $is_replicated;
+
+    if (PVE::QemuServer::check_running($source_vmid)) {
+        $add_error->('offline-migration-vm-running', "cannot migrate running VM without --online")
+            if !$param->{online};
+    } else {
+        $add_warning->(
+            'online-migration-vm-not-running',
+            "VM isn't running, doing offline migration instead",
+        ) if $param->{online};
+        $param->{online} = 0;
+    }
+
+    my $check_custom_cpu = sub {
+        return if !defined($conf->{cpu});
+
+        my $cpu = PVE::JSONSchema::parse_property_string('pve-vm-cpu-conf', $conf->{cpu});
+        my $cputype = $cpu->{cputype};
+        return if !defined($cputype) || !PVE::QemuServer::CPUConfig::is_custom_model($cputype);
+
+        my $custom_cpu = PVE::QemuServer::CPUConfig::get_custom_model($cputype);
+
+        my $remote_custom_cpu = eval {
+            $api_client->get(
+                "/cluster/qemu/custom-cpu-models/" . URI::Escape::uri_escape_utf8($cputype));
+        };
+
+        if ($@) {
+            $add_error->(
+                'custom-cpu-validation',
+                "could not validate custom CPU model compatibility: $@",
+            );
+            return;
+        }
+
+        my $cpu_schema = {
+            type => 'object',
+            properties => PVE::QemuServer::CPUConfig->options(),
+        };
+        eval { PVE::JSONSchema::validate($remote_custom_cpu, $cpu_schema); };
+
+        if ($@) {
+            $add_error->(
+                'custom-cpu-validation',
+                "could not validate custom CPU model compatibility: $@",
+            );
+            return;
+        }
+
+        eval {
+            PVE::QemuServer::CPUConfig::assert_custom_model_compatibility(
+                $custom_cpu, $remote_custom_cpu,
+            );
+        };
+
+        $add_error->('custom-cpu-mismatch', $@) if $@;
+    };
+    $check_custom_cpu->();
+
+    $add_error->('storage-mapping', "remote migration requires explicit storage mapping")
+        if $storagemap->{identity};
+
+    return {
+        conn_args => $conn_args,
+        api_client => $api_client,
+        source_vmid => $source_vmid,
+        target_vmid => $target_vmid,
+        storagemap => $storagemap,
+        bridgemap => $bridgemap,
+        delete => $delete,
+    };
+}
+
 __PACKAGE__->register_method({
     name => 'vmlist',
     path => '',
@@ -5713,104 +5902,26 @@ __PACKAGE__->register_method({
         my $rpcenv = PVE::RPCEnvironment::get();
         my $authuser = $rpcenv->get_user();
 
-        my $source_vmid = extract_param($param, 'vmid');
-        my $target_endpoint = extract_param($param, 'target-endpoint');
-        my $target_vmid = extract_param($param, 'target-vmid') // $source_vmid;
-
-        my $delete = extract_param($param, 'delete') // 0;
-
-        PVE::Cluster::check_cfs_quorum();
-
-        # test if VM exists
-        my $conf = PVE::QemuConfig->load_config($source_vmid);
-
-        PVE::QemuConfig->check_lock($conf);
-
-        raise_param_exc({ vmid => "cannot remote-migrate VM that is configured for HA" })
-            if PVE::HA::Config::service_is_configured("vm:$source_vmid");
-
-        my $remote = PVE::JSONSchema::parse_property_string('proxmox-remote', $target_endpoint);
-
-        # TODO: move this as helper somewhere appropriate?
-        my $conn_args = {
-            protocol => 'https',
-            host => $remote->{host},
-            port => $remote->{port} // 8006,
-            apitoken => $remote->{apitoken},
-        };
-
-        if ($remote->{fingerprint}) {
-            $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
-        }
-
-        print "Establishing API connection with remote at '$remote->{host}'\n";
-
-        my $api_client = PVE::APIClient::LWP->new(%$conn_args);
-
-        my $repl_conf = PVE::ReplicationConfig->new();
-        my $is_replicated = $repl_conf->check_for_existing_jobs($source_vmid, 1);
-        die "cannot remote-migrate replicated VM\n" if $is_replicated;
+        my $findings = [];
+        my $plan = validate_remote_migrate_preconditions($param, $findings);
 
-        if (PVE::QemuServer::check_running($source_vmid)) {
-            die "can't migrate running VM without --online\n" if !$param->{online};
-
-        } else {
-            warn "VM isn't running. Doing offline migration instead.\n" if $param->{online};
-            $param->{online} = 0;
+        foreach my $finding (@$findings) {
+            warn "$finding->{message}\n" if $finding->{severity} eq 'warning';
+            die "$finding->{message}\n" if $finding->{severity} eq 'error';
         }
 
-        if (defined($conf->{cpu})) {
-            my $cpu = PVE::JSONSchema::parse_property_string('pve-vm-cpu-conf', $conf->{cpu});
-            my $cputype = $cpu->{cputype};
-            if (defined($cputype) && PVE::QemuServer::CPUConfig::is_custom_model($cputype)) {
-                my $custom_cpu = PVE::QemuServer::CPUConfig::get_custom_model($cputype);
-
-                my $remote_custom_cpu = eval {
-                    $api_client->get("/cluster/qemu/custom-cpu-models/"
-                        . URI::Escape::uri_escape_utf8($cputype));
-                };
-                die "could not validate custom CPU model compatibility: $@\n" if $@;
-
-                my $cpu_schema = {
-                    type => 'object',
-                    properties => PVE::QemuServer::CPUConfig->options(),
-                };
-                eval { PVE::JSONSchema::validate($remote_custom_cpu, $cpu_schema); };
-                die "could not validate custom CPU model compatibility: $@\n" if $@;
-
-                PVE::QemuServer::CPUConfig::assert_custom_model_compatibility(
-                    $custom_cpu, $remote_custom_cpu,
-                );
-            }
-        }
-
-        my $storecfg = PVE::Storage::config();
-        my $target_storage = extract_param($param, 'target-storage');
-        my $storagemap =
-            eval { PVE::JSONSchema::parse_idmap($target_storage, 'pve-storage-id') };
-        raise_param_exc({ 'target-storage' => "failed to parse storage map: $@" })
-            if $@;
-
-        my $target_bridge = extract_param($param, 'target-bridge');
-        my $bridgemap = eval { PVE::JSONSchema::parse_idmap($target_bridge, 'pve-bridge-id') };
-        raise_param_exc({ 'target-bridge' => "failed to parse bridge map: $@" })
-            if $@;
-
-        die "remote migration requires explicit storage mapping!\n"
-            if $storagemap->{identity};
-
-        $param->{storagemap} = $storagemap;
-        $param->{bridgemap} = $bridgemap;
+        $param->{storagemap} = $plan->{storagemap};
+        $param->{bridgemap} = $plan->{bridgemap};
         $param->{remote} = {
-            conn => $conn_args, # re-use fingerprint for tunnel
-            client => $api_client,
-            vmid => $target_vmid,
+            conn => $plan->{conn_args}, # re-use fingerprint for tunnel
+            client => $plan->{api_client},
+            vmid => $plan->{target_vmid},
         };
         $param->{migration_type} = 'websocket';
         $param->{'with-local-disks'} = 1;
-        $param->{delete} = $delete if $delete;
+        $param->{delete} = $plan->{delete} if $plan->{delete};
 
-        my $cluster_status = $api_client->get("/cluster/status");
+        my $cluster_status = $plan->{api_client}->get("/cluster/status");
         my $target_node;
         foreach my $entry (@$cluster_status) {
             next if $entry->{type} ne 'node';
@@ -5824,14 +5935,21 @@ __PACKAGE__->register_method({
             if !defined($target_node);
 
         my $realcmd = sub {
-            PVE::QemuMigrate->migrate($target_node, $remote->{host}, $source_vmid, $param);
+            PVE::QemuMigrate->migrate(
+                $target_node,
+                $plan->{conn_args}->{host},
+                $plan->{source_vmid},
+                $param,
+            );
         };
 
         my $worker = sub {
-            return PVE::GuestHelpers::guest_migration_lock($source_vmid, 10, $realcmd);
+            return PVE::GuestHelpers::guest_migration_lock($plan->{source_vmid}, 10, $realcmd);
         };
 
-        return $rpcenv->fork_worker('qmigrate', $source_vmid, $authuser, $worker);
+        return $rpcenv->fork_worker(
+            'qmigrate', $plan->{source_vmid}, $authuser, $worker,
+        );
     },
 });
 
-- 
2.47.3

[RFC qemu-server 3/4] qm: remote-migrate: call API endpoint directly

[Erik Fastermann]

The command wrapper only ran a subset of the precondition checks the
endpoint now performs itself. Register the 'remote-migrate' command
against PVE::API2::Qemu directly, as done for 'migrate' and the other
commands, and drop the wrapper.

Also remove the now unused PVE::APIClient::LWP import.

Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
---
 src/PVE/CLI/qm.pm | 128 +---------------------------------------------
 1 file changed, 1 insertion(+), 127 deletions(-)

diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm
index b903c1f1..ec87972c 100755
--- a/src/PVE/CLI/qm.pm
+++ b/src/PVE/CLI/qm.pm
@@ -15,7 +15,6 @@ use POSIX qw(strftime);
 use Term::ReadLine;
 use URI::Escape;
 
-use PVE::APIClient::LWP;
 use PVE::Cluster;
 use PVE::Exception qw(raise_param_exc);
 use PVE::GuestHelpers;
@@ -172,131 +171,6 @@ __PACKAGE__->register_method({
     },
 });
 
-__PACKAGE__->register_method({
-    name => 'remote_migrate_vm',
-    path => 'remote_migrate_vm',
-    method => 'POST',
-    description =>
-        "Migrate virtual machine to a remote cluster. Creates a new migration task. EXPERIMENTAL feature!",
-    permissions => {
-        check => ['perm', '/vms/{vmid}', ['VM.Migrate']],
-    },
-    parameters => {
-        additionalProperties => 0,
-        properties => {
-            node => get_standard_option('pve-node'),
-            vmid =>
-                get_standard_option('pve-vmid', { completion => \&PVE::QemuServer::complete_vmid }),
-            'target-vmid' => get_standard_option('pve-vmid', { optional => 1 }),
-            'target-endpoint' => get_standard_option('proxmox-remote', {
-                    description => "Remote target endpoint",
-            }),
-            online => {
-                type => 'boolean',
-                description =>
-                    "Use online/live migration if VM is running. Ignored if VM is stopped.",
-                optional => 1,
-            },
-            delete => {
-                type => 'boolean',
-                description =>
-                    "Delete the original VM and related data after successful migration. By default the original VM is kept on the source cluster in a stopped state.",
-                optional => 1,
-                default => 0,
-            },
-            'target-storage' => get_standard_option(
-                'pve-targetstorage',
-                {
-                    completion => \&PVE::QemuServer::complete_migration_storage,
-                    optional => 0,
-                },
-            ),
-            'target-bridge' => {
-                type => 'string',
-                description =>
-                    "Mapping from source to target bridges. Providing only a single bridge ID maps all source bridges to that bridge. Providing the special value '1' will map each source bridge to itself.",
-                format => 'bridge-pair-list',
-            },
-            bwlimit => {
-                description => "Override I/O bandwidth limit (in KiB/s).",
-                optional => 1,
-                type => 'integer',
-                minimum => '0',
-                default => 'migrate limit from datacenter or storage config',
-            },
-        },
-    },
-    returns => {
-        type => 'string',
-        description => "the task ID.",
-    },
-    code => sub {
-        my ($param) = @_;
-
-        my $rpcenv = PVE::RPCEnvironment::get();
-        my $authuser = $rpcenv->get_user();
-
-        my $source_vmid = $param->{vmid};
-        my $target_endpoint = $param->{'target-endpoint'};
-        my $target_vmid = $param->{'target-vmid'} // $source_vmid;
-
-        my $remote = PVE::JSONSchema::parse_property_string('proxmox-remote', $target_endpoint);
-
-        # TODO: move this as helper somewhere appropriate?
-        my $conn_args = {
-            protocol => 'https',
-            host => $remote->{host},
-            port => $remote->{port} // 8006,
-            apitoken => $remote->{apitoken},
-        };
-
-        $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 }
-            if defined($remote->{fingerprint});
-
-        my $api_client = PVE::APIClient::LWP->new(%$conn_args);
-        my $resources = $api_client->get("/cluster/resources", { type => 'vm' });
-        if (grep { defined($_->{vmid}) && $_->{vmid} eq $target_vmid } @$resources) {
-            raise_param_exc(
-                {
-                    target_vmid =>
-                        "Guest with ID '$target_vmid' already exists on remote cluster",
-                },
-            );
-        }
-
-        my $storages = $api_client->get("/nodes/localhost/storage", { enabled => 1 });
-
-        my $storecfg = PVE::Storage::config();
-        my $target_storage = $param->{'target-storage'};
-        my $storagemap =
-            eval { PVE::JSONSchema::parse_idmap($target_storage, 'pve-storage-id') };
-        raise_param_exc({ 'target-storage' => "failed to parse storage map: $@" })
-            if $@;
-
-        my $check_remote_storage = sub {
-            my ($storage) = @_;
-            my $found = [grep { $_->{storage} eq $storage } @$storages];
-            die "remote: storage '$storage' does not exist (or missing permission)!\n"
-                if !@$found;
-
-            $found = @$found[0];
-
-            my $content_types = [PVE::Tools::split_list($found->{content})];
-            die "remote: storage '$storage' cannot store images\n"
-                if !grep { $_ eq 'images' } @$content_types;
-        };
-
-        foreach my $target_sid (values %{ $storagemap->{entries} }) {
-            $check_remote_storage->($target_sid);
-        }
-
-        $check_remote_storage->($storagemap->{default})
-            if $storagemap->{default};
-
-        return PVE::API2::Qemu->remote_migrate_vm($param);
-    },
-});
-
 __PACKAGE__->register_method({
     name => 'status',
     path => 'status',
@@ -1361,7 +1235,7 @@ our $cmddef = {
 
     migrate => ["PVE::API2::Qemu", 'migrate_vm', ['vmid', 'target'], {%node}, $upid_exit],
     'remote-migrate' => [
-        __PACKAGE__,
+        "PVE::API2::Qemu",
         'remote_migrate_vm',
         ['vmid', 'target-vmid', 'target-endpoint'],
         {%node},
-- 
2.47.3

[RFC qemu-server 4/4] remote migrate: add precondition check endpoint

[Erik Fastermann]

Add a remote_migrate_vm_precondition endpoint that runs the remote
migration precondition checks without starting a migration and returns
them as a list of {severity, code, message} findings, so callers such
as the web UI can surface blockers and warnings before the user commits
to a migration.

It reuses the same validate_remote_migrate_preconditions helper the
migrate endpoint uses, so the precheck cannot drift from what is
actually enforced. The endpoint is POST rather than GET, on its own
path, to keep the remote API token out of request URLs and the access
log.

Suggested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
---
 src/PVE/API2/Qemu.pm | 156 ++++++++++++++++++++++++++++++-------------
 1 file changed, 111 insertions(+), 45 deletions(-)

diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 9b487457..ade1451e 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -1042,6 +1042,53 @@ sub assert_scsi_feature_compatibility {
     }
 }
 
+my $remote_migrate_vm_properties = {
+    additionalProperties => 0,
+    properties => {
+        node => get_standard_option('pve-node'),
+        vmid =>
+            get_standard_option('pve-vmid', { completion => \&PVE::QemuServer::complete_vmid }),
+        'target-vmid' => get_standard_option('pve-vmid', { optional => 1 }),
+        'target-endpoint' => get_standard_option('proxmox-remote', {
+                description => "Remote target endpoint",
+        }),
+        online => {
+            type => 'boolean',
+            description =>
+                "Use online/live migration if VM is running. Ignored if VM is stopped.",
+            optional => 1,
+        },
+        delete => {
+            type => 'boolean',
+            description => "Delete the original VM and related data after successful migration."
+                . " By default the original VM is kept on the source cluster in a stopped state.",
+            optional => 1,
+            default => 0,
+        },
+        'target-storage' => get_standard_option(
+            'pve-targetstorage',
+            {
+                completion => \&PVE::QemuServer::complete_migration_storage,
+                optional => 0,
+            },
+        ),
+        'target-bridge' => {
+            type => 'string',
+            description => "Mapping from source to target bridges. Providing only a single"
+                . " bridge ID maps all source bridges to that bridge. Providing the special"
+                . " value '1' will map each source bridge to itself.",
+            format => 'bridge-pair-list',
+        },
+        bwlimit => {
+            description => "Override I/O bandwidth limit (in KiB/s).",
+            optional => 1,
+            type => 'integer',
+            minimum => '0',
+            default => 'migrate limit from datacenter or storage config',
+        },
+    },
+};
+
 my sub validate_remote_migrate_preconditions {
     my ($param, $findings) = @_;
 
@@ -5837,61 +5884,80 @@ __PACKAGE__->register_method({
 });
 
 __PACKAGE__->register_method({
-    name => 'remote_migrate_vm',
-    path => '{vmid}/remote_migrate',
+    name => 'remote_migrate_vm_precondition',
+    path => '{vmid}/remote_migrate_precondition',
     method => 'POST',
     protected => 1,
     proxyto => 'node',
-    description =>
-        "Migrate virtual machine to a remote cluster. Creates a new migration task. EXPERIMENTAL feature!",
+    description => "Get preconditions for remote migration.",
     permissions => {
         check => ['perm', '/vms/{vmid}', ['VM.Migrate']],
     },
-    parameters => {
-        additionalProperties => 0,
-        properties => {
-            node => get_standard_option('pve-node'),
-            vmid =>
-                get_standard_option('pve-vmid', { completion => \&PVE::QemuServer::complete_vmid }),
-            'target-vmid' => get_standard_option('pve-vmid', { optional => 1 }),
-            'target-endpoint' => get_standard_option('proxmox-remote', {
-                    description => "Remote target endpoint",
-            }),
-            online => {
-                type => 'boolean',
-                description =>
-                    "Use online/live migration if VM is running. Ignored if VM is stopped.",
-                optional => 1,
-            },
-            delete => {
-                type => 'boolean',
-                description =>
-                    "Delete the original VM and related data after successful migration. By default the original VM is kept on the source cluster in a stopped state.",
-                optional => 1,
-                default => 0,
-            },
-            'target-storage' => get_standard_option(
-                'pve-targetstorage',
-                {
-                    completion => \&PVE::QemuServer::complete_migration_storage,
-                    optional => 0,
+    parameters => { $remote_migrate_vm_properties->%* },
+    returns => {
+        type => "array",
+        items => {
+            description => "List of findings.",
+            type => "object",
+            properties => {
+                severity => {
+                    description => "Severity of the finding.",
+                    type => 'string',
+                    enum => [qw(error warning)],
                 },
-            ),
-            'target-bridge' => {
-                type => 'string',
-                description =>
-                    "Mapping from source to target bridges. Providing only a single bridge ID maps all source bridges to that bridge. Providing the special value '1' will map each source bridge to itself.",
-                format => 'bridge-pair-list',
-            },
-            bwlimit => {
-                description => "Override I/O bandwidth limit (in KiB/s).",
-                optional => 1,
-                type => 'integer',
-                minimum => '0',
-                default => 'migrate limit from datacenter or storage config',
+                code => {
+                    description => "Machine readable code of the finding.",
+                    type => 'string',
+                    enum => [qw(
+                        load-vm-config
+                        vm-locked
+                        vm-ha-configured
+                        remote-conn
+                        target-vmid-exists
+                        storage-missing
+                        storage-no-images
+                        vm-replicated
+                        offline-migration-vm-running
+                        online-migration-vm-not-running
+                        custom-cpu-validation
+                        custom-cpu-mismatch
+                        storage-mapping
+                    )],
+                },
+                message => {
+                    description => "Human readable message.",
+                    type => 'string',
+                },
+                storage => get_standard_option(
+                    'pve-storage-id',
+                    {
+                        description => "Optional associated storage.",
+                        optional => 1,
+                    },
+                ),
             },
         },
     },
+    code => sub {
+        my ($param) = @_;
+        my $findings = [];
+        validate_remote_migrate_preconditions($param, $findings);
+        return $findings;
+    },
+});
+
+__PACKAGE__->register_method({
+    name => 'remote_migrate_vm',
+    path => '{vmid}/remote_migrate',
+    method => 'POST',
+    protected => 1,
+    proxyto => 'node',
+    description =>
+        "Migrate virtual machine to a remote cluster. Creates a new migration task. EXPERIMENTAL feature!",
+    permissions => {
+        check => ['perm', '/vms/{vmid}', ['VM.Migrate']],
+    },
+    parameters => { $remote_migrate_vm_properties->%* },
     returns => {
         type => 'string',
         description => "the task ID.",
-- 
2.47.3

Re: [RFC qemu-server 2/4] remote migrate: collect preconditions as structured findings

[Fiona Ebner]

Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
> Extract the remote-migration precondition checks into a
> validate_remote_migrate_preconditions helper that records each problem
> as a {severity, code, message} finding and returns the derived
> migration parameters. The endpoint fails on the first error finding,
> preserving the previous abort behavior.
> 
> It also pulls the checks that previously lived only in the qm CLI into
> the endpoint. Direct API callers such as the web UI now run them too,
> so fewer migrations start only to fail partway. This prepares for a
> precondition endpoint that returns the full findings list instead of
> dying on the first error.

Nit: that part could be it's own patch

> User-visible effect is minimal: successful migrations are unchanged.
> Some precondition error messages are reworded and the "Establishing
> API connection" info line is no longer printed.

Do we gain anything from not printing it? Such changes can also be nicer
in separate patches to ease reviewability and discussion.

> 
> Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
> ---
>  src/PVE/API2/Qemu.pm | 304 ++++++++++++++++++++++++++++++-------------
>  1 file changed, 211 insertions(+), 93 deletions(-)
> 
> diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
> index 68f1800c..9b487457 100644
> --- a/src/PVE/API2/Qemu.pm
> +++ b/src/PVE/API2/Qemu.pm
> @@ -1042,6 +1042,195 @@ sub assert_scsi_feature_compatibility {
>      }
>  }
>  
> +my sub validate_remote_migrate_preconditions {
> +    my ($param, $findings) = @_;
> +
> +    my $add_error = sub {

Style nit: please collect the subroutine arguments as named arguments
like usual. In particular, you can use a hash like %extra_info as the
last one.

> +        push @$findings,
> +            {
> +                severity => 'error',
> +                code => $_[0],

Nit: for the {storage}/import-metadata endpoint we use 'type' rather
than 'code'.

> +                message => ("$_[1]" =~ s/\s+$//r),

Please use chomp() rather than such a regex (or PVE::Tools::trim, but I
don't think there'll be trailing whitespace, else we should rather fix
it where the messsage originates).

> +                @_[2 .. $#_],
> +            };
> +    };
> +
> +    my $add_warning = sub {

Style nit: please collect the subroutine arguments as named arguments
like usual.

> +        push @$findings,
> +            {
> +                severity => 'warning',
> +                code => $_[0],
> +                message => ("$_[1]" =~ s/\s+$//r),

Same as above

> +                @_[2 .. $#_],
> +            };
> +    };
> +
> +    my $source_vmid = extract_param($param, 'vmid');
> +    my $target_endpoint = extract_param($param, 'target-endpoint');
> +    my $remote = PVE::JSONSchema::parse_property_string('proxmox-remote', $target_endpoint);
> +    my $target_vmid = extract_param($param, 'target-vmid') // $source_vmid;
> +
> +    my $target_storage = extract_param($param, 'target-storage');
> +    my $storagemap = eval { PVE::JSONSchema::parse_idmap($target_storage, 'pve-storage-id') };
> +    raise_param_exc({ 'target-storage' => "failed to parse storage map: $@" })
> +        if $@;
> +
> +    my $target_bridge = extract_param($param, 'target-bridge');
> +    my $bridgemap = eval { PVE::JSONSchema::parse_idmap($target_bridge, 'pve-bridge-id') };
> +    raise_param_exc({ 'target-bridge' => "failed to parse bridge map: $@" })
> +        if $@;
> +
> +    my $delete = extract_param($param, 'delete') // 0;
> +
> +    PVE::Cluster::check_cfs_quorum();
> +
> +    # test if VM exists
> +    my $conf = eval { PVE::QemuConfig->load_config($source_vmid) };
> +    if ($@) {

Style nit: please use if (my $err = $@) { to immediately assign the
error to a variable. Same for the other instances.

> +        $add_error->('load-vm-config', $@);
> +        return;
> +    }
> +
> +    if (PVE::QemuConfig->has_lock($conf)) {
> +        $add_error->('vm-locked', "VM is locked ($conf->{lock})");
> +    }
> +
> +    if (PVE::HA::Config::service_is_configured("vm:$source_vmid")) {
> +        $add_error->('vm-ha-configured', 'cannot remote-migrate VM that is configured for HA');
> +    }
> +
> +    my $conn_args = {
> +        protocol => 'https',
> +        host => $remote->{host},
> +        port => $remote->{port} // 8006,
> +        apitoken => $remote->{apitoken},
> +    };
> +
> +    if ($remote->{fingerprint}) {
> +        $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
> +    }
> +
> +    my $api_client = PVE::APIClient::LWP->new(%$conn_args);
> +
> +    # check connection once at the start
> +    eval { $api_client->get("/version") };
> +    if ($@) {
> +        $add_error->('remote-conn', $@);
> +        return;
> +    }
> +
> +    my $resources = $api_client->get("/cluster/resources", { type => 'vm' });
> +    if (grep { defined($_->{vmid}) && $_->{vmid} eq $target_vmid } @$resources) {
> +        $add_error->('target-vmid-exists', "remote: guest with ID '$target_vmid' already exists");
> +    }
> +
> +    my $storages = $api_client->get("/nodes/localhost/storage", { enabled => 1 });
> +
> +    my $check_remote_storage = sub {
> +        my ($storage) = @_;
> +        my $found = [grep { $_->{storage} eq $storage } @$storages];
> +
> +        if (!@$found) {
> +            $add_error->(
> +                'storage-missing',
> +                "remote: storage '$storage' does not exist (or missing permission)",
> +                storage => $storage,
> +            );
> +            return;
> +        }
> +
> +        $found = $found->[0];
> +        my $content_types = [PVE::Tools::split_list($found->{content})];
> +        $add_error->(
> +            'storage-no-images',
> +            "remote: storage '$storage' cannot store images",
> +            storage => $storage,
> +        ) if !grep { $_ eq 'images' } @$content_types;
> +    };
> +
> +    foreach my $target_sid (values %{ $storagemap->{entries} }) {

Style nit: please use for instead of foreach, please use post deref ->%*

> +        $check_remote_storage->($target_sid);
> +    }
> +

---snip 8<---

> @@ -5713,104 +5902,26 @@ __PACKAGE__->register_method({
>          my $rpcenv = PVE::RPCEnvironment::get();
>          my $authuser = $rpcenv->get_user();
>  
> -        my $source_vmid = extract_param($param, 'vmid');
> -        my $target_endpoint = extract_param($param, 'target-endpoint');
> -        my $target_vmid = extract_param($param, 'target-vmid') // $source_vmid;
> -
> -        my $delete = extract_param($param, 'delete') // 0;
> -
> -        PVE::Cluster::check_cfs_quorum();
> -
> -        # test if VM exists
> -        my $conf = PVE::QemuConfig->load_config($source_vmid);
> -
> -        PVE::QemuConfig->check_lock($conf);
> -
> -        raise_param_exc({ vmid => "cannot remote-migrate VM that is configured for HA" })
> -            if PVE::HA::Config::service_is_configured("vm:$source_vmid");
> -
> -        my $remote = PVE::JSONSchema::parse_property_string('proxmox-remote', $target_endpoint);
> -
> -        # TODO: move this as helper somewhere appropriate?
> -        my $conn_args = {
> -            protocol => 'https',
> -            host => $remote->{host},
> -            port => $remote->{port} // 8006,
> -            apitoken => $remote->{apitoken},
> -        };
> -
> -        if ($remote->{fingerprint}) {
> -            $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
> -        }
> -
> -        print "Establishing API connection with remote at '$remote->{host}'\n";
> -
> -        my $api_client = PVE::APIClient::LWP->new(%$conn_args);
> -
> -        my $repl_conf = PVE::ReplicationConfig->new();
> -        my $is_replicated = $repl_conf->check_for_existing_jobs($source_vmid, 1);
> -        die "cannot remote-migrate replicated VM\n" if $is_replicated;
> +        my $findings = [];
> +        my $plan = validate_remote_migrate_preconditions($param, $findings);

Style nit: why pass in $findings like this? You can just return multiple
values instead. I'd prefer a name like $migration_info rather than $plan.

>  
> -        if (PVE::QemuServer::check_running($source_vmid)) {
> -            die "can't migrate running VM without --online\n" if !$param->{online};
> -
> -        } else {
> -            warn "VM isn't running. Doing offline migration instead.\n" if $param->{online};
> -            $param->{online} = 0;
> +        foreach my $finding (@$findings) {

Style nit: please use for instead of foreach

> +            warn "$finding->{message}\n" if $finding->{severity} eq 'warning';
> +            die "$finding->{message}\n" if $finding->{severity} eq 'error';

We can print all warnings and errors and then die with a message saying
how many warnings and errors there are.

>          }
>  
> -        if (defined($conf->{cpu})) {
> -            my $cpu = PVE::JSONSchema::parse_property_string('pve-vm-cpu-conf', $conf->{cpu});
> -            my $cputype = $cpu->{cputype};
> -            if (defined($cputype) && PVE::QemuServer::CPUConfig::is_custom_model($cputype)) {
> -                my $custom_cpu = PVE::QemuServer::CPUConfig::get_custom_model($cputype);
> -
> -                my $remote_custom_cpu = eval {
> -                    $api_client->get("/cluster/qemu/custom-cpu-models/"
> -                        . URI::Escape::uri_escape_utf8($cputype));
> -                };
> -                die "could not validate custom CPU model compatibility: $@\n" if $@;
> -
> -                my $cpu_schema = {
> -                    type => 'object',
> -                    properties => PVE::QemuServer::CPUConfig->options(),
> -                };
> -                eval { PVE::JSONSchema::validate($remote_custom_cpu, $cpu_schema); };
> -                die "could not validate custom CPU model compatibility: $@\n" if $@;
> -
> -                PVE::QemuServer::CPUConfig::assert_custom_model_compatibility(
> -                    $custom_cpu, $remote_custom_cpu,
> -                );
> -            }
> -        }
> -
> -        my $storecfg = PVE::Storage::config();
> -        my $target_storage = extract_param($param, 'target-storage');
> -        my $storagemap =
> -            eval { PVE::JSONSchema::parse_idmap($target_storage, 'pve-storage-id') };
> -        raise_param_exc({ 'target-storage' => "failed to parse storage map: $@" })
> -            if $@;
> -
> -        my $target_bridge = extract_param($param, 'target-bridge');
> -        my $bridgemap = eval { PVE::JSONSchema::parse_idmap($target_bridge, 'pve-bridge-id') };
> -        raise_param_exc({ 'target-bridge' => "failed to parse bridge map: $@" })
> -            if $@;
> -
> -        die "remote migration requires explicit storage mapping!\n"
> -            if $storagemap->{identity};
> -
> -        $param->{storagemap} = $storagemap;
> -        $param->{bridgemap} = $bridgemap;
> +        $param->{storagemap} = $plan->{storagemap};
> +        $param->{bridgemap} = $plan->{bridgemap};
>          $param->{remote} = {
> -            conn => $conn_args, # re-use fingerprint for tunnel
> -            client => $api_client,
> -            vmid => $target_vmid,
> +            conn => $plan->{conn_args}, # re-use fingerprint for tunnel
> +            client => $plan->{api_client},
> +            vmid => $plan->{target_vmid},
>          };
>          $param->{migration_type} = 'websocket';
>          $param->{'with-local-disks'} = 1;
> -        $param->{delete} = $delete if $delete;
> +        $param->{delete} = $plan->{delete} if $plan->{delete};
>  
> -        my $cluster_status = $api_client->get("/cluster/status");
> +        my $cluster_status = $plan->{api_client}->get("/cluster/status");
>          my $target_node;
>          foreach my $entry (@$cluster_status) {
>              next if $entry->{type} ne 'node';
> @@ -5824,14 +5935,21 @@ __PACKAGE__->register_method({
>              if !defined($target_node);
>  
>          my $realcmd = sub {
> -            PVE::QemuMigrate->migrate($target_node, $remote->{host}, $source_vmid, $param);
> +            PVE::QemuMigrate->migrate(
> +                $target_node,
> +                $plan->{conn_args}->{host},
> +                $plan->{source_vmid},
> +                $param,
> +            );
>          };
>  
>          my $worker = sub {
> -            return PVE::GuestHelpers::guest_migration_lock($source_vmid, 10, $realcmd);
> +            return PVE::GuestHelpers::guest_migration_lock($plan->{source_vmid}, 10, $realcmd);
>          };
>  
> -        return $rpcenv->fork_worker('qmigrate', $source_vmid, $authuser, $worker);
> +        return $rpcenv->fork_worker(
> +            'qmigrate', $plan->{source_vmid}, $authuser, $worker,
> +        );
>      },
>  });
>  

Re: [RFC qemu-server 4/4] remote migrate: add precondition check endpoint

[Fiona Ebner]

Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
> +                storage => get_standard_option(
> +                    'pve-storage-id',
> +                    {
> +                        description => "Optional associated storage.",
> +                        optional => 1,
> +                    },
> +                ),

Is the idea to include the storage ID to represent such findings
differently from other warnings/errors in the UI?

Re: [RFC qemu-server 0/4] remote migrate: extract preconditions and add check endpoint

[Fiona Ebner]

Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
> This series reworks how the QEMU remote-migration precondition checks
> are structured and exposes them through a new endpoint, so blockers and
> warnings can be surfaced before a migration is started rather than
> mid-flight.
> 
> It is deliberately sent as an RFC: the change touches a critical path,
> and the larger direction (see the open questions below) matters more
> than the current diff. Thanks to @Fabian Gruenbichler for the discussion
> so far.
> 
> Motivation
> ==========
> 
> Today the checks run at the very start of a remote migration and abort
> on the first error via die. Two problems follow:
> 
> - Many prerequisites are not checked up front at all; they only surface
>   once the migration is already running, e.g. local/mapped devices or a
>   VNC clipboard that is not live-migratable. A user fixes one blocker,
>   retries, and hits the next. A forum user collected a checklist of such
>   prerequisites [0] (thanks @Arthur Bied-Charreton for pointing this
>   out).
> 
> - The qm CLI wrapper only ran a subset of the checks the API path ran,
>   so direct API callers (e.g. the web UI) and the CLI disagreed on what
>   was validated.
> 
> What the series does
> ====================
> 
> 1. Drop the ineffective fingerprint auto-detection.
> 
> 2. Extract the checks into a validate_remote_migrate_preconditions
>    helper.
> 
> 3. Register the remote-migrate command against PVE::API2::Qemu
>    directly and drop the CLI wrapper, so CLI and API run the exact same
>    checks.
> 
> 4. Add a remote_migrate_vm_precondition endpoint that runs the checks
>    without starting a migration and returns the full findings list. It
>    reuses the same helper as the migrate endpoint, so the precheck
>    cannot drift from what is actually enforced.
> 
> The precondition results are returned as a flat list of findings. Does
> this shape look reasonable, and could a similar structure be reused for
> the intra-cluster migration precondition endpoint?

For the intra-cluster endpoint we also collect information for each
node, not just for a single target.

> 
> Open questions (RFC)
> ====================
> 
> Scope of the checks
> - The checks should stay semi-static, i.e. mostly config-derived, and
>   run quickly, i.e. avoid slow probes such as querying storage
>   contents. But we still want to query the remote, since a precheck
>   that never talks to the target is not very useful. Where/how can we
>   reasonably reduce the number of API round trips to the remote?

I think it can be done as part of the tunnel, see below. But this can be
done after (or on top of your current series).

> 
> Where should the checks run?
> - The source node currently runs the checks with a few local and fast
>   remote API calls. Since we want migrations between major versions to
>   work in a mostly forward- and backward-compatible way (older source
>   to newer target being the typical case), it might be cleaner to run
>   the checks on the remote by sending the relevant local config over.
>   When the remote is newer, it can also run more or patched checks.
>   Of course this would be a larger change, as the general structure
>   currently goes from source to target.

Yes, I would prefer the source passing the relevant info and have all
checks that can be done by the target to be done by the target.

> - Collecting results from both sides in PDM and comparing there seems
>   wrong: the checks still have to run locally for the real migration,
>   and both paths would then need to agree. We should rather keep a
>   single authority.

Yes, PDM is not a prerequisite for remote migration, so it should not be
dependent on PDM. The source should do its own checks and call into the
endpoint for the target side.

> - Running the checks over the Tunnel command instead would add a new
>   command that older endpoints do not understand. Both here and for the
>   API, a capability/feature endpoint could help: source and/or target
>   query what each supports for a migration, and older versions fall
>   back to the current checks. Is that worth introducing now?

The capability negotiation can be done via a new tunnel command, see:
https://lore.proxmox.com/pve-devel/b072f9d7-b0f8-41fc-8c37-7ea90798aa1b@proxmox.com/

It should be possible to issue the capabilities command to the target
and detect if the target side understands it (look at the error message
to see if unknown or failed for other reasons) and otherwise it's too
old and has no capabilities at all.

> 
> Two-phase refactor
> - The cleaner fix for "checks only run mid-migration" is to extract all
>   preconditions and split execution into two phases. Phase 1 is shared
>   by the precondition endpoint and the migration: it runs all the
>   reasonable checks and builds a plan hash of the queried info. Phase 2
>   performs the actual migration, ideally with its remaining checks up
>   front and ordered fast-to-slow. This is a large refactor of a
>   critical path, as the cluster migration code is mostly shared with
>   the remote migration. In addition, the cluster migration precondition
>   endpoint can also be refactored, which is an open TODO. Is that the
>   direction we want?

In the long term checking as much as possible up-front would be best,
yes. But some checks are a bit more involved to do up-front (e.g.
storage migration format negotiation also needs a brush-up). For the
current series, it's better to focus on things that are sensible/easier
to do. The rest can follow, once we have the mechanism in place. I don't
think we should drop existing checks from the migration path, or even
can, because they are still relevant for a source/target without the
precondition capability.

> 
> Containers
> - A similar series could later cover remote container migration. What
>   pitfalls should I watch out for there?

I don't think there's a fundamental difference with the design. In fact,
it should be simpler, because there is no live migration, so there are
fewer scenarios.

> 
> References
> ==========
> [0] https://forum.proxmox.com/threads/pdm-cross-cluster-migration-prerequisites-checklist-i-wish-i-had-before-my-first-attempt.184485
> 
> Thanks in advance for the feedback!
> 
> 
> Erik Fastermann (4):
>   remote migrate: drop ineffective fingerprint auto-detection
>   remote migrate: collect preconditions as structured findings
>   qm: remote-migrate: call API endpoint directly
>   remote migrate: add precondition check endpoint
> 
>  src/PVE/API2/Qemu.pm | 472 +++++++++++++++++++++++++++++--------------
>  src/PVE/CLI/qm.pm    | 128 +-----------
>  2 files changed, 323 insertions(+), 277 deletions(-)
> 

Re: [RFC qemu-server 1/4] remote migrate: drop ineffective fingerprint auto-detection

[Fiona Ebner]

Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
> When no fingerprint was supplied, the code fetched the remote node
> certificate and pinned its fingerprint for the migration tunnel. This
> has no functional effect and is removed.
> 
> That fetch only succeeds for remotes trusted via the CA store, which
> is exactly the case where pinning is unnecessary: both the API client
> and the websocket tunnel fall back to CA verification on their own.
> For a self-signed remote the fetch itself fails verification, so no
> fingerprint is ever obtained.

The websocket tunnel checks the pinned fingerprint against the one that
was supplied, so with the current code any weird scenario where the
fingerprint queried from the certificates API endpoint does not match
the one gotten later by the websocket tunnel connection is caught. If
you remove the pinning, such scenarios won't be caught anymore. Not sure
how important that is though.

@Fabian what do you think?

> 
> The only meaningful path, an explicitly supplied fingerprint (needed to
> verify self-signed remotes), is unchanged.
> 
> Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
> ---
>  src/PVE/API2/Qemu.pm | 16 ++--------------
>  1 file changed, 2 insertions(+), 14 deletions(-)
> 
> diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
> index 28cbb9b0..68f1800c 100644
> --- a/src/PVE/API2/Qemu.pm
> +++ b/src/PVE/API2/Qemu.pm
> @@ -5739,26 +5739,14 @@ __PACKAGE__->register_method({
>              apitoken => $remote->{apitoken},
>          };
>  
> -        my $fp;
> -        if ($fp = $remote->{fingerprint}) {
> -            $conn_args->{cached_fingerprints} = { uc($fp) => 1 };
> +        if ($remote->{fingerprint}) {
> +            $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
>          }
>  
>          print "Establishing API connection with remote at '$remote->{host}'\n";
>  
>          my $api_client = PVE::APIClient::LWP->new(%$conn_args);
>  
> -        if (!defined($fp)) {
> -            my $cert_info = $api_client->get("/nodes/localhost/certificates/info");
> -            foreach my $cert (@$cert_info) {
> -                my $filename = $cert->{filename};
> -                next if $filename ne 'pveproxy-ssl.pem' && $filename ne 'pve-ssl.pem';
> -                $fp = $cert->{fingerprint} if !$fp || $filename eq 'pveproxy-ssl.pem';
> -            }
> -            $conn_args->{cached_fingerprints} = { uc($fp) => 1 }
> -                if defined($fp);
> -        }
> -
>          my $repl_conf = PVE::ReplicationConfig->new();
>          my $is_replicated = $repl_conf->check_for_existing_jobs($source_vmid, 1);
>          die "cannot remote-migrate replicated VM\n" if $is_replicated;

Re: [RFC qemu-server 1/4] remote migrate: drop ineffective fingerprint auto-detection

[Fiona Ebner]

Forgot to CC Fabian, doing it now :)

Am 29.07.26 um 11:57 AM schrieb Fiona Ebner:
> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>> When no fingerprint was supplied, the code fetched the remote node
>> certificate and pinned its fingerprint for the migration tunnel. This
>> has no functional effect and is removed.
>>
>> That fetch only succeeds for remotes trusted via the CA store, which
>> is exactly the case where pinning is unnecessary: both the API client
>> and the websocket tunnel fall back to CA verification on their own.
>> For a self-signed remote the fetch itself fails verification, so no
>> fingerprint is ever obtained.
> 
> The websocket tunnel checks the pinned fingerprint against the one that
> was supplied, so with the current code any weird scenario where the
> fingerprint queried from the certificates API endpoint does not match
> the one gotten later by the websocket tunnel connection is caught. If
> you remove the pinning, such scenarios won't be caught anymore. Not sure
> how important that is though.
> 
> @Fabian what do you think?
> 
>>
>> The only meaningful path, an explicitly supplied fingerprint (needed to
>> verify self-signed remotes), is unchanged.
>>
>> Signed-off-by: Erik Fastermann <e.fastermann@proxmox.com>
>> ---
>>  src/PVE/API2/Qemu.pm | 16 ++--------------
>>  1 file changed, 2 insertions(+), 14 deletions(-)
>>
>> diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
>> index 28cbb9b0..68f1800c 100644
>> --- a/src/PVE/API2/Qemu.pm
>> +++ b/src/PVE/API2/Qemu.pm
>> @@ -5739,26 +5739,14 @@ __PACKAGE__->register_method({
>>              apitoken => $remote->{apitoken},
>>          };
>>  
>> -        my $fp;
>> -        if ($fp = $remote->{fingerprint}) {
>> -            $conn_args->{cached_fingerprints} = { uc($fp) => 1 };
>> +        if ($remote->{fingerprint}) {
>> +            $conn_args->{cached_fingerprints} = { uc($remote->{fingerprint}) => 1 };
>>          }
>>  
>>          print "Establishing API connection with remote at '$remote->{host}'\n";
>>  
>>          my $api_client = PVE::APIClient::LWP->new(%$conn_args);
>>  
>> -        if (!defined($fp)) {
>> -            my $cert_info = $api_client->get("/nodes/localhost/certificates/info");
>> -            foreach my $cert (@$cert_info) {
>> -                my $filename = $cert->{filename};
>> -                next if $filename ne 'pveproxy-ssl.pem' && $filename ne 'pve-ssl.pem';
>> -                $fp = $cert->{fingerprint} if !$fp || $filename eq 'pveproxy-ssl.pem';
>> -            }
>> -            $conn_args->{cached_fingerprints} = { uc($fp) => 1 }
>> -                if defined($fp);
>> -        }
>> -
>>          my $repl_conf = PVE::ReplicationConfig->new();
>>          my $is_replicated = $repl_conf->check_for_existing_jobs($source_vmid, 1);
>>          die "cannot remote-migrate replicated VM\n" if $is_replicated;
> 
> 
> 
> 
> 

Re: [RFC qemu-server 1/4] remote migrate: drop ineffective fingerprint auto-detection

[Fiona Ebner]

Am 29.07.26 um 11:59 AM schrieb Fiona Ebner:
> Forgot to CC Fabian, doing it now :)

Or actually I did not, but the list doesn't show it :P

Re: [RFC qemu-server 1/4] remote migrate: drop ineffective fingerprint auto-detection

[Fabian Grünbichler]

On July 29, 2026 11:56 am, Fiona Ebner wrote:
> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>> When no fingerprint was supplied, the code fetched the remote node
>> certificate and pinned its fingerprint for the migration tunnel. This
>> has no functional effect and is removed.
>> 
>> That fetch only succeeds for remotes trusted via the CA store, which
>> is exactly the case where pinning is unnecessary: both the API client
>> and the websocket tunnel fall back to CA verification on their own.
>> For a self-signed remote the fetch itself fails verification, so no
>> fingerprint is ever obtained.
> 
> The websocket tunnel checks the pinned fingerprint against the one that
> was supplied, so with the current code any weird scenario where the
> fingerprint queried from the certificates API endpoint does not match
> the one gotten later by the websocket tunnel connection is caught. If
> you remove the pinning, such scenarios won't be caught anymore. Not sure
> how important that is though.
> 
> @Fabian what do you think?

Erik actually asked me about this before sending the RFC (@Erik - feel
free to mention such things in the "notes" part of a patch).

I don't think it really buys us much, and AFAICT it breaks in the
following case with a certificate trusted by the system:

- certificate info queried
- cert rotated
- websocket tunnel established -> wrong fingerprint

because the websocket tunnel will only check the fingerprint if one is
provided.

so IMHO, this should be removed.

Re: [RFC qemu-server 1/4] remote migrate: drop ineffective fingerprint auto-detection

[Fiona Ebner]

Am 29.07.26 um 12:06 PM schrieb Fabian Grünbichler:
> On July 29, 2026 11:56 am, Fiona Ebner wrote:
>> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>>> When no fingerprint was supplied, the code fetched the remote node
>>> certificate and pinned its fingerprint for the migration tunnel. This
>>> has no functional effect and is removed.
>>>
>>> That fetch only succeeds for remotes trusted via the CA store, which
>>> is exactly the case where pinning is unnecessary: both the API client
>>> and the websocket tunnel fall back to CA verification on their own.
>>> For a self-signed remote the fetch itself fails verification, so no
>>> fingerprint is ever obtained.
>>
>> The websocket tunnel checks the pinned fingerprint against the one that
>> was supplied, so with the current code any weird scenario where the
>> fingerprint queried from the certificates API endpoint does not match
>> the one gotten later by the websocket tunnel connection is caught. If
>> you remove the pinning, such scenarios won't be caught anymore. Not sure
>> how important that is though.
>>
>> @Fabian what do you think?
> 
> Erik actually asked me about this before sending the RFC (@Erik - feel
> free to mention such things in the "notes" part of a patch).
> 
> I don't think it really buys us much, and AFAICT it breaks in the
> following case with a certificate trusted by the system:
> 
> - certificate info queried
> - cert rotated
> - websocket tunnel established -> wrong fingerprint
> 
> because the websocket tunnel will only check the fingerprint if one is
> provided.
> 
> so IMHO, this should be removed.

Ack, sounds good :)

Re: [RFC qemu-server 0/4] remote migrate: extract preconditions and add check endpoint

[Erik Fastermann]

Thank you Fiona for the review. Replies inline.


> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>> This series reworks how the QEMU remote-migration precondition checks
>> are structured and exposes them through a new endpoint, so blockers and
>> warnings can be surfaced before a migration is started rather than
>> mid-flight.
>>
>> It is deliberately sent as an RFC: the change touches a critical path,
>> and the larger direction (see the open questions below) matters more
>> than the current diff. Thanks to @Fabian Gruenbichler for the discussion
>> so far.
>>
>> Motivation
>> ==========
>>
>> Today the checks run at the very start of a remote migration and abort
>> on the first error via die. Two problems follow:
>>
>> - Many prerequisites are not checked up front at all; they only surface
>>    once the migration is already running, e.g. local/mapped devices or a
>>    VNC clipboard that is not live-migratable. A user fixes one blocker,
>>    retries, and hits the next. A forum user collected a checklist of such
>>    prerequisites [0] (thanks @Arthur Bied-Charreton for pointing this
>>    out).
>>
>> - The qm CLI wrapper only ran a subset of the checks the API path ran,
>>    so direct API callers (e.g. the web UI) and the CLI disagreed on what
>>    was validated.
>>
>> What the series does
>> ====================
>>
>> 1. Drop the ineffective fingerprint auto-detection.
>>
>> 2. Extract the checks into a validate_remote_migrate_preconditions
>>     helper.
>>
>> 3. Register the remote-migrate command against PVE::API2::Qemu
>>     directly and drop the CLI wrapper, so CLI and API run the exact same
>>     checks.
>>
>> 4. Add a remote_migrate_vm_precondition endpoint that runs the checks
>>     without starting a migration and returns the full findings list. It
>>     reuses the same helper as the migrate endpoint, so the precheck
>>     cannot drift from what is actually enforced.
>>
>> The precondition results are returned as a flat list of findings. Does
>> this shape look reasonable, and could a similar structure be reused for
>> the intra-cluster migration precondition endpoint?
> 
> For the intra-cluster endpoint we also collect information for each
> node, not just for a single target.
> 

Right, I phrased that badly: I meant collecting the findings per node, 
so the shape would be a map from node to findings list rather than a 
single list.

One argument for the flat-list shape that I should have made up front: 
It looks like the web UI already reduces the endpoint output to this 
shape in window/Migrate.js which builds migration.preconditions, a list 
of { text, severity: error|warning } objects.

Reworking the intra-cluster endpoint is out of scope for this series 
though. I only want the findings shape designed so it can be reused 
there later.

>>
>> Open questions (RFC)
>> ====================
>>
>> Scope of the checks
>> - The checks should stay semi-static, i.e. mostly config-derived, and
>>    run quickly, i.e. avoid slow probes such as querying storage
>>    contents. But we still want to query the remote, since a precheck
>>    that never talks to the target is not very useful. Where/how can we
>>    reasonably reduce the number of API round trips to the remote?
> 
> I think it can be done as part of the tunnel, see below. But this can be
> done after (or on top of your current series).
> 

See below under the capability point.

>>
>> Where should the checks run?
>> - The source node currently runs the checks with a few local and fast
>>    remote API calls. Since we want migrations between major versions to
>>    work in a mostly forward- and backward-compatible way (older source
>>    to newer target being the typical case), it might be cleaner to run
>>    the checks on the remote by sending the relevant local config over.
>>    When the remote is newer, it can also run more or patched checks.
>>    Of course this would be a larger change, as the general structure
>>    currently goes from source to target.
> 
> Yes, I would prefer the source passing the relevant info and have all
> checks that can be done by the target to be done by the target.
> 

Agreed, that is what the plan under the capability point below does.


>> - Collecting results from both sides in PDM and comparing there seems
>>    wrong: the checks still have to run locally for the real migration,
>>    and both paths would then need to agree. We should rather keep a
>>    single authority.
> 
> Yes, PDM is not a prerequisite for remote migration, so it should not be
> dependent on PDM. The source should do its own checks and call into the
> endpoint for the target side.
> 

Agreed.

>> - Running the checks over the Tunnel command instead would add a new
>>    command that older endpoints do not understand. Both here and for the
>>    API, a capability/feature endpoint could help: source and/or target
>>    query what each supports for a migration, and older versions fall
>>    back to the current checks. Is that worth introducing now?
> 
> The capability negotiation can be done via a new tunnel command, see:
> https://lore.proxmox.com/pve-devel/b072f9d7-b0f8-41fc-8c37-7ea90798aa1b@proxmox.com/
> 
> It should be possible to issue the capabilities command to the target
> and detect if the target side understands it (look at the error message
> to see if unknown or failed for other reasons) and otherwise it's too
> old and has no capabilities at all.
> 

That direction makes sense to me. Concretely:

- Add a capabilities tunnel command, plus a precondition tunnel command 
in which the source sends the relevant config and the target runs the 
checks it can answer better than the source can.

- The existing source-side checks stay. They are still the only thing 
that runs against a target without the capability, so removing them 
would mean no checks at all for new -> old migrations.

- New checks that need target-side knowledge go through the new command; 
new checks that are purely source-config-derived go into the shared 
helper this series adds.

- The new tunnel commands are issued by the precondition endpoint.

- That leaves the current series structurally as-is, with your feedback 
applied, and the tunnel work follows on top.

What do you think about this?

>>
>> Two-phase refactor
>> - The cleaner fix for "checks only run mid-migration" is to extract all
>>    preconditions and split execution into two phases. Phase 1 is shared
>>    by the precondition endpoint and the migration: it runs all the
>>    reasonable checks and builds a plan hash of the queried info. Phase 2
>>    performs the actual migration, ideally with its remaining checks up
>>    front and ordered fast-to-slow. This is a large refactor of a
>>    critical path, as the cluster migration code is mostly shared with
>>    the remote migration. In addition, the cluster migration precondition
>>    endpoint can also be refactored, which is an open TODO. Is that the
>>    direction we want?
> 
> In the long term checking as much as possible up-front would be best,
> yes. But some checks are a bit more involved to do up-front (e.g.
> storage migration format negotiation also needs a brush-up). For the
> current series, it's better to focus on things that are sensible/easier
> to do. The rest can follow, once we have the mechanism in place. I don't
> think we should drop existing checks from the migration path, or even
> can, because they are still relevant for a source/target without the
> precondition capability.
> 

Agreed on all counts, including keeping the existing checks in the 
migration path, see above.

The motivation for extracting checks was less about the two-phase split 
itself and more about not implementing the same rule twice to prevent 
drift between the precondition and migration endpoint. It is important 
that we implement all simple checks in a way that the precondition 
endpoint uses the same code as the migration endpoint.

>>
>> Containers
>> - A similar series could later cover remote container migration. What
>>    pitfalls should I watch out for there?
> 
> I don't think there's a fundamental difference with the design. In fact,
> it should be simpler, because there is no live migration, so there are
> fewer scenarios.
> 

Good, then I can tackle that after this has settled.

>>
>> References
>> ==========
>> [0] https://forum.proxmox.com/threads/pdm-cross-cluster-migration-prerequisites-checklist-i-wish-i-had-before-my-first-attempt.184485
>>
>> Thanks in advance for the feedback!
>>
>>
>> Erik Fastermann (4):
>>    remote migrate: drop ineffective fingerprint auto-detection
>>    remote migrate: collect preconditions as structured findings
>>    qm: remote-migrate: call API endpoint directly
>>    remote migrate: add precondition check endpoint
>>
>>   src/PVE/API2/Qemu.pm | 472 +++++++++++++++++++++++++++++--------------
>>   src/PVE/CLI/qm.pm    | 128 +-----------
>>   2 files changed, 323 insertions(+), 277 deletions(-)
>>
> 

Re: [RFC qemu-server 2/4] remote migrate: collect preconditions as structured findings

[Erik Fastermann]

[snip]

>> User-visible effect is minimal: successful migrations are unchanged.
>> Some precondition error messages are reworded and the "Establishing
>> API connection" info line is no longer printed.
> 
> Do we gain anything from not printing it? Such changes can also be nicer
> in separate patches to ease reviewability and discussion.
> 

I thought printing the info line would be noise in the precondition 
endpoint, which I can put in the commit message, in a separate patch.

[snip]
>> +        my $findings = [];
>> +        my $plan = validate_remote_migrate_preconditions($param, $findings);
> 
> Style nit: why pass in $findings like this? You can just return multiple
> values instead. I'd prefer a name like $migration_info rather than $plan.
> 

I didn't want to return the findings array on every possible exit point 
of the function, so I thought this would be nicer. I could add an extra 
wrapper function which does this though, so I will try that.

[snip]

Re: [RFC qemu-server 4/4] remote migrate: add precondition check endpoint

[Erik Fastermann]

> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>> +                storage => get_standard_option(
>> +                    'pve-storage-id',
>> +                    {
>> +                        description => "Optional associated storage.",
>> +                        optional => 1,
>> +                    },
>> +                ),
> 
> Is the idea to include the storage ID to represent such findings
> differently from other warnings/errors in the UI?

The field is meant as an extension of the code field for machine 
consumers, so a caller can tell which object a finding refers to without 
parsing the message. The UI will most likely need it: The message we 
return cannot always be translated AFAIK, so the UI has to build its own 
string from the code and needs the storage ID separately to fill it in.

Storage is currently the only such field, because it is the only one the 
existing checks need. Further fields can be added the same way as more 
checks arrive.

Re: [RFC qemu-server 0/4] remote migrate: extract preconditions and add check endpoint

[Daniel Kral]

On Fri Jul 31, 2026 at 3:42 PM CEST, Erik Fastermann wrote:
>> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>>> This series reworks how the QEMU remote-migration precondition checks
>>> are structured and exposes them through a new endpoint, so blockers and
>>> warnings can be surfaced before a migration is started rather than
>>> mid-flight.
>>>
>>> It is deliberately sent as an RFC: the change touches a critical path,
>>> and the larger direction (see the open questions below) matters more
>>> than the current diff. Thanks to @Fabian Gruenbichler for the discussion
>>> so far.
>>>
>>> Motivation
>>> ==========
>>>
>>> Today the checks run at the very start of a remote migration and abort
>>> on the first error via die. Two problems follow:
>>>
>>> - Many prerequisites are not checked up front at all; they only surface
>>>    once the migration is already running, e.g. local/mapped devices or a
>>>    VNC clipboard that is not live-migratable. A user fixes one blocker,
>>>    retries, and hits the next. A forum user collected a checklist of such
>>>    prerequisites [0] (thanks @Arthur Bied-Charreton for pointing this
>>>    out).
>>>
>>> - The qm CLI wrapper only ran a subset of the checks the API path ran,
>>>    so direct API callers (e.g. the web UI) and the CLI disagreed on what
>>>    was validated.
>>>
>>> What the series does
>>> ====================
>>>
>>> 1. Drop the ineffective fingerprint auto-detection.
>>>
>>> 2. Extract the checks into a validate_remote_migrate_preconditions
>>>     helper.
>>>
>>> 3. Register the remote-migrate command against PVE::API2::Qemu
>>>     directly and drop the CLI wrapper, so CLI and API run the exact same
>>>     checks.
>>>
>>> 4. Add a remote_migrate_vm_precondition endpoint that runs the checks
>>>     without starting a migration and returns the full findings list. It
>>>     reuses the same helper as the migrate endpoint, so the precheck
>>>     cannot drift from what is actually enforced.
>>>
>>> The precondition results are returned as a flat list of findings. Does
>>> this shape look reasonable, and could a similar structure be reused for
>>> the intra-cluster migration precondition endpoint?
>> 
>> For the intra-cluster endpoint we also collect information for each
>> node, not just for a single target.
>> 
>
> Right, I phrased that badly: I meant collecting the findings per node, 
> so the shape would be a map from node to findings list rather than a 
> single list.
>
> One argument for the flat-list shape that I should have made up front: 
> It looks like the web UI already reduces the endpoint output to this 
> shape in window/Migrate.js which builds migration.preconditions, a list 
> of { text, severity: error|warning } objects.
>
> Reworking the intra-cluster endpoint is out of scope for this series 
> though. I only want the findings shape designed so it can be reused 
> there later.

Just as a heads-up: As part of deriving node affinity rules from the
intra-cluster migration blockers / constraints, I'm working on a patch
series for the intra-cluster part at the moment.

This is done to infer such constraints automatically for HA resources,
but allows us to broaden some CRS functionalities as the load balancing
mechanism to non-HA resources as well without burdening users to encode
all these constraints manually as affinity rules.

Though there we need the functionality to derive these constraints for
all guests (VMs and LXCs) and all cluster nodes at once and need to know
if any of the required configs have changed so we don't need to
reevaluate all these cluster-wide constraint checks every CRM round.

So I suppose the structure for these might be quite different from the
ones for the intra-cluster checks, but as you said this is out-of-scope
of your patch series, I just wanted to give a heads-up :)