Re: [RFC qemu-server 0/4] remote migrate: extract preconditions and add check endpoint

[Erik Fastermann <e.fastermann@proxmox.com>]

Thank you Fiona for the review. Replies inline.


> Am 21.07.26 um 1:58 PM schrieb Erik Fastermann:
>> This series reworks how the QEMU remote-migration precondition checks
>> are structured and exposes them through a new endpoint, so blockers and
>> warnings can be surfaced before a migration is started rather than
>> mid-flight.
>>
>> It is deliberately sent as an RFC: the change touches a critical path,
>> and the larger direction (see the open questions below) matters more
>> than the current diff. Thanks to @Fabian Gruenbichler for the discussion
>> so far.
>>
>> Motivation
>> ==========
>>
>> Today the checks run at the very start of a remote migration and abort
>> on the first error via die. Two problems follow:
>>
>> - Many prerequisites are not checked up front at all; they only surface
>>    once the migration is already running, e.g. local/mapped devices or a
>>    VNC clipboard that is not live-migratable. A user fixes one blocker,
>>    retries, and hits the next. A forum user collected a checklist of such
>>    prerequisites [0] (thanks @Arthur Bied-Charreton for pointing this
>>    out).
>>
>> - The qm CLI wrapper only ran a subset of the checks the API path ran,
>>    so direct API callers (e.g. the web UI) and the CLI disagreed on what
>>    was validated.
>>
>> What the series does
>> ====================
>>
>> 1. Drop the ineffective fingerprint auto-detection.
>>
>> 2. Extract the checks into a validate_remote_migrate_preconditions
>>     helper.
>>
>> 3. Register the remote-migrate command against PVE::API2::Qemu
>>     directly and drop the CLI wrapper, so CLI and API run the exact same
>>     checks.
>>
>> 4. Add a remote_migrate_vm_precondition endpoint that runs the checks
>>     without starting a migration and returns the full findings list. It
>>     reuses the same helper as the migrate endpoint, so the precheck
>>     cannot drift from what is actually enforced.
>>
>> The precondition results are returned as a flat list of findings. Does
>> this shape look reasonable, and could a similar structure be reused for
>> the intra-cluster migration precondition endpoint?
> 
> For the intra-cluster endpoint we also collect information for each
> node, not just for a single target.
> 

Right, I phrased that badly: I meant collecting the findings per node, 
so the shape would be a map from node to findings list rather than a 
single list.

One argument for the flat-list shape that I should have made up front: 
It looks like the web UI already reduces the endpoint output to this 
shape in window/Migrate.js which builds migration.preconditions, a list 
of { text, severity: error|warning } objects.

Reworking the intra-cluster endpoint is out of scope for this series 
though. I only want the findings shape designed so it can be reused 
there later.

>>
>> Open questions (RFC)
>> ====================
>>
>> Scope of the checks
>> - The checks should stay semi-static, i.e. mostly config-derived, and
>>    run quickly, i.e. avoid slow probes such as querying storage
>>    contents. But we still want to query the remote, since a precheck
>>    that never talks to the target is not very useful. Where/how can we
>>    reasonably reduce the number of API round trips to the remote?
> 
> I think it can be done as part of the tunnel, see below. But this can be
> done after (or on top of your current series).
> 

See below under the capability point.

>>
>> Where should the checks run?
>> - The source node currently runs the checks with a few local and fast
>>    remote API calls. Since we want migrations between major versions to
>>    work in a mostly forward- and backward-compatible way (older source
>>    to newer target being the typical case), it might be cleaner to run
>>    the checks on the remote by sending the relevant local config over.
>>    When the remote is newer, it can also run more or patched checks.
>>    Of course this would be a larger change, as the general structure
>>    currently goes from source to target.
> 
> Yes, I would prefer the source passing the relevant info and have all
> checks that can be done by the target to be done by the target.
> 

Agreed, that is what the plan under the capability point below does.


>> - Collecting results from both sides in PDM and comparing there seems
>>    wrong: the checks still have to run locally for the real migration,
>>    and both paths would then need to agree. We should rather keep a
>>    single authority.
> 
> Yes, PDM is not a prerequisite for remote migration, so it should not be
> dependent on PDM. The source should do its own checks and call into the
> endpoint for the target side.
> 

Agreed.

>> - Running the checks over the Tunnel command instead would add a new
>>    command that older endpoints do not understand. Both here and for the
>>    API, a capability/feature endpoint could help: source and/or target
>>    query what each supports for a migration, and older versions fall
>>    back to the current checks. Is that worth introducing now?
> 
> The capability negotiation can be done via a new tunnel command, see:
> https://lore.proxmox.com/pve-devel/b072f9d7-b0f8-41fc-8c37-7ea90798aa1b@proxmox.com/
> 
> It should be possible to issue the capabilities command to the target
> and detect if the target side understands it (look at the error message
> to see if unknown or failed for other reasons) and otherwise it's too
> old and has no capabilities at all.
> 

That direction makes sense to me. Concretely:

- Add a capabilities tunnel command, plus a precondition tunnel command 
in which the source sends the relevant config and the target runs the 
checks it can answer better than the source can.

- The existing source-side checks stay. They are still the only thing 
that runs against a target without the capability, so removing them 
would mean no checks at all for new -> old migrations.

- New checks that need target-side knowledge go through the new command; 
new checks that are purely source-config-derived go into the shared 
helper this series adds.

- The new tunnel commands are issued by the precondition endpoint.

- That leaves the current series structurally as-is, with your feedback 
applied, and the tunnel work follows on top.

What do you think about this?

>>
>> Two-phase refactor
>> - The cleaner fix for "checks only run mid-migration" is to extract all
>>    preconditions and split execution into two phases. Phase 1 is shared
>>    by the precondition endpoint and the migration: it runs all the
>>    reasonable checks and builds a plan hash of the queried info. Phase 2
>>    performs the actual migration, ideally with its remaining checks up
>>    front and ordered fast-to-slow. This is a large refactor of a
>>    critical path, as the cluster migration code is mostly shared with
>>    the remote migration. In addition, the cluster migration precondition
>>    endpoint can also be refactored, which is an open TODO. Is that the
>>    direction we want?
> 
> In the long term checking as much as possible up-front would be best,
> yes. But some checks are a bit more involved to do up-front (e.g.
> storage migration format negotiation also needs a brush-up). For the
> current series, it's better to focus on things that are sensible/easier
> to do. The rest can follow, once we have the mechanism in place. I don't
> think we should drop existing checks from the migration path, or even
> can, because they are still relevant for a source/target without the
> precondition capability.
> 

Agreed on all counts, including keeping the existing checks in the 
migration path, see above.

The motivation for extracting checks was less about the two-phase split 
itself and more about not implementing the same rule twice to prevent 
drift between the precondition and migration endpoint. It is important 
that we implement all simple checks in a way that the precondition 
endpoint uses the same code as the migration endpoint.

>>
>> Containers
>> - A similar series could later cover remote container migration. What
>>    pitfalls should I watch out for there?
> 
> I don't think there's a fundamental difference with the design. In fact,
> it should be simpler, because there is no live migration, so there are
> fewer scenarios.
> 

Good, then I can tackle that after this has settled.

>>
>> References
>> ==========
>> [0] https://forum.proxmox.com/threads/pdm-cross-cluster-migration-prerequisites-checklist-i-wish-i-had-before-my-first-attempt.184485
>>
>> Thanks in advance for the feedback!
>>
>>
>> Erik Fastermann (4):
>>    remote migrate: drop ineffective fingerprint auto-detection
>>    remote migrate: collect preconditions as structured findings
>>    qm: remote-migrate: call API endpoint directly
>>    remote migrate: add precondition check endpoint
>>
>>   src/PVE/API2/Qemu.pm | 472 +++++++++++++++++++++++++++++--------------
>>   src/PVE/CLI/qm.pm    | 128 +-----------
>>   2 files changed, 323 insertions(+), 277 deletions(-)
>>
>