Re: [pve-devel] [RFC] qemu-server: add migration_type=insecure to remote-migrate

["Fabian Grünbichler" <f.gruenbichler@proxmox.com>]

On April 25, 2026 3:10 am, Bogdan Ionescu wrote:
> Hi all,
> 
> I'd like to gauge interest in adding a migration_type=insecure option to
> the qm remote-migrate / remote_migrate_vm endpoint, before investing
> time in a review-ready patch series.

Hi!

This is something that we will need sooner or later as well, in the
context of PDM and fabrics.

> == Motivation ==
> 
> The current remote-migrate implementation tunnels both control plane
> and data plane through the websocket connection to the target's API
> endpoint on 8006/tcp. This is the right default for trust reasons
> (API token + TLS fingerprint, no SSH trust between clusters needed),
> but the data plane throughput is severely bottlenecked by:
> 
>   - userspace bouncing through PVE::Tunnel + pveproxy + qmtunnel
>     (3 Perl processes in the data path, each context-switching per
>     chunk)
>   - per-byte WebSocket masking in pure Perl (RFC 6455 §5.3)
>   - TLS framing on top
>   - lack of zero-copy / TSO offload for the streamed bytes
>   - multiple TCP segments end-to-end with independent flow control
> 
> In our deployment between two DCs connected by WireGuard over a
> 10 Gbps link, we observe sustained ~1 MB/s for remote-migrate while
> intra-cluster `qm migrate --migration_type insecure` between the same
> hosts saturates the link at ~300+ MB/s. The bottleneck is clearly
> the WS tunnel data path on a single Perl-bound core, not the network.
> 
> For VMs with 32+ GB of RAM, this difference is the difference between
> a migration finishing in 2 minutes vs. failing to converge entirely
> because the dirty rate exceeds the throughput.
> 
> == Proposal ==
> 
> Mirror the local-cluster migration model: keep secure (WS-tunneled) as
> the default, allow opt-in 'insecure' for trusted networks where the
> operator has out-of-band guarantees (private cross-connect, VPN,
> overlay encryption at L2/L3).
> 
>   qm remote-migrate <vmid> <target-vmid> 'apitoken=...,host=...,fp=...' \
>       --target-storage ... --target-bridge ... --online \
>       --migration_type insecure \
>       --migration_network 10.50.0.0/24
> 
> Semantics:
>   - control plane (config, NBD allocation requests, tunnel commands,
>     spice ticket, etc.) still goes through the WS tunnel as today

this makes sense

>   - data plane (QEMU memory stream + NBD storage drive-mirror) goes
>     direct TCP between source and target on the standard
>     60000-60050 range, with the target's listener IP resolved from
>     --migration_network (same logic as local-cluster insecure)

this as well, though as an alternative one might consider providing an
interface name as well?

>   - root-only on the source side, consistent with migrate_vm

here we have a slight difference between intra-cluster and inter-cluster
migrations:
- within a cluster, we have established trust and a shared
  authentication scope - node A asking node B about its migration
  address is okay (post-authentication), since a regular user cannot
  override it
- between clusters, we have less guarantees - while the target has to
  trust the source somewhat (which is why we require a separate
  privilege for allowing incoming remote migrations in the first place),
  I am not sure whether we would not want to require some additional
  privileges for allowing insecure migrations as well? e.g. Sys.Modify
  somewhere, or something similar?

we might also consider whether it makes sense to pre-configure remote
migration networks and allow selecting them by ID, though that could be
added later as follow-up as well.

>   - target advertises an 'insecure-remote' capability in the mtunnel
>     version response so source can fail closed on older targets

right, without this an outdated remote node would start the VM with tcp
migration, but the mtunnel endpoint would then die because it only
allows unix sockets atm..

> 
> == Backward compatibility approach ==
> 
> Rather than bumping WS_TUNNEL_VERSION (which would break
> new-source -> old-target combinations because of the existing
> "$WS_TUNNEL_VERSION > $tunnel->{version}" check), I'd add a
> forward-compatible 'caps' field to the version response. Old sources
> ignore unknown JSON keys; new sources require 'insecure-remote' to be
> present in caps before allowing migration_type=insecure, and otherwise
> fall through to the existing WS-tunneled path with no behavioral
> change.

what do you think @Fiona?

> This means all four mix matrices are clean:
>   - patched <-> patched, secure: identical to today
>   - unpatched src -> patched tgt: caps ignored, WS path as today
>   - patched src -> unpatched tgt, secure: caps absent, not checked,
>     WS path as today
>   - patched src -> unpatched tgt, insecure: source dies early with a
>     clear "upgrade target or omit migration_type=insecure" error,
>     no side effects on target
> 
> == Security considerations ==
> 
>   - root-only at the API/CLI layer, same as the local-cluster knob
>   - documented as requiring trusted/private network between clusters

this part here is a big one - it really needs documentation that screams
"double check to ensure this doesn't accidentally broadcast clear text
migration data over the internet"

>   - no change to control plane or auth (still API token + TLS fp)
>   - data plane confidentiality drops to network-layer controls only,
>     which is identical to the trade-off operators already make for
>     intra-cluster insecure migration
>   - no new ports beyond the existing 60000-60050 range that
>     insecure migration already uses
>   - source-side caps check ensures no silent downgrade when target
>     doesn't support it
> 
> == Open questions ==
> 
>   1. Is this direction acceptable in principle, or would you prefer
>      a different direction?

it mostly looks good to me with a quick glance, it might be sensible to
wait for additional input by Fiona or Thomas before diving in.

>   2. Should the 'caps' mechanism be added in a standalone preliminary
>      patch (useful as future-proofing for any opt-in mtunnel feature),
>      or rolled into the same series?
> 
>   3. Should NBD direct-TCP be gated by a separate flag, or is it fine
>      to have migration_type=insecure imply both RAM and NBD direct?
>      The intra-cluster knob ties them together today.

I think it's fine to tie them together, the same considerations apply to
them.

>   4. Any preference on the parameter name? I matched migrate_vm
>      ('migration_type', 'migration_network') for consistency, but
>      'data-direct-tcp' or similar would also work and arguably be
>      less misleading since the control plane is still encrypted.

that's also true for local migration - the data plane is SSH in that
case..