Re: [pve-devel] [RFC] qemu-server: add migration_type=insecure to remote-migrate

[Bogdan Ionescu <bogdan@ionescu.at>]

Hi,

following the feedback, I reworked the implementation and split it into a small patch series.

Changes compared to the RFC version:

kept the websocket tunnel for the control plane only
migration and NBD storage traffic use direct TCP only when migration_type=insecure
added capability-based negotiation (insecure-remote) instead of bumping the tunnel API version
added support for migration_type / migration_network to qm remote-migrate
added additional permission checks requiring Sys.Modify on /
added more explicit warnings that guest RAM and disk migration data may be transferred unencrypted

I tested this with online remote migration between two PVE 9 nodes using local ZFS-backed disks.

The remote migration path successfully used:

NBD over direct TCP for storage migration
direct TCP for QEMU migration state
websocket tunnel only for the control plane

Patch series follows.

Kind regards, 
Bogdan

On Thursday, April 30th, 2026 at 3:40 PM, Fabian Grünbichler <f.gruenbichler@proxmox.com> wrote:

> On April 25, 2026 3:10 am, Bogdan Ionescu wrote:
> > Hi all,
> >
> > I'd like to gauge interest in adding a migration_type=insecure option to
> > the qm remote-migrate / remote_migrate_vm endpoint, before investing
> > time in a review-ready patch series.
> 
> Hi!
> 
> This is something that we will need sooner or later as well, in the
> context of PDM and fabrics.
> 
> > == Motivation ==
> >
> > The current remote-migrate implementation tunnels both control plane
> > and data plane through the websocket connection to the target's API
> > endpoint on 8006/tcp. This is the right default for trust reasons
> > (API token + TLS fingerprint, no SSH trust between clusters needed),
> > but the data plane throughput is severely bottlenecked by:
> >
> >   - userspace bouncing through PVE::Tunnel + pveproxy + qmtunnel
> >     (3 Perl processes in the data path, each context-switching per
> >     chunk)
> >   - per-byte WebSocket masking in pure Perl (RFC 6455 §5.3)
> >   - TLS framing on top
> >   - lack of zero-copy / TSO offload for the streamed bytes
> >   - multiple TCP segments end-to-end with independent flow control
> >
> > In our deployment between two DCs connected by WireGuard over a
> > 10 Gbps link, we observe sustained ~1 MB/s for remote-migrate while
> > intra-cluster `qm migrate --migration_type insecure` between the same
> > hosts saturates the link at ~300+ MB/s. The bottleneck is clearly
> > the WS tunnel data path on a single Perl-bound core, not the network.
> >
> > For VMs with 32+ GB of RAM, this difference is the difference between
> > a migration finishing in 2 minutes vs. failing to converge entirely
> > because the dirty rate exceeds the throughput.
> >
> > == Proposal ==
> >
> > Mirror the local-cluster migration model: keep secure (WS-tunneled) as
> > the default, allow opt-in 'insecure' for trusted networks where the
> > operator has out-of-band guarantees (private cross-connect, VPN,
> > overlay encryption at L2/L3).
> >
> >   qm remote-migrate <vmid> <target-vmid> 'apitoken=...,host=...,fp=...' \
> >       --target-storage ... --target-bridge ... --online \
> >       --migration_type insecure \
> >       --migration_network 10.50.0.0/24
> >
> > Semantics:
> >   - control plane (config, NBD allocation requests, tunnel commands,
> >     spice ticket, etc.) still goes through the WS tunnel as today
> 
> this makes sense
> 
> >   - data plane (QEMU memory stream + NBD storage drive-mirror) goes
> >     direct TCP between source and target on the standard
> >     60000-60050 range, with the target's listener IP resolved from
> >     --migration_network (same logic as local-cluster insecure)
> 
> this as well, though as an alternative one might consider providing an
> interface name as well?
> 
> >   - root-only on the source side, consistent with migrate_vm
> 
> here we have a slight difference between intra-cluster and inter-cluster
> migrations:
> - within a cluster, we have established trust and a shared
>   authentication scope - node A asking node B about its migration
>   address is okay (post-authentication), since a regular user cannot
>   override it
> - between clusters, we have less guarantees - while the target has to
>   trust the source somewhat (which is why we require a separate
>   privilege for allowing incoming remote migrations in the first place),
>   I am not sure whether we would not want to require some additional
>   privileges for allowing insecure migrations as well? e.g. Sys.Modify
>   somewhere, or something similar?
> 
> we might also consider whether it makes sense to pre-configure remote
> migration networks and allow selecting them by ID, though that could be
> added later as follow-up as well.
> 
> >   - target advertises an 'insecure-remote' capability in the mtunnel
> >     version response so source can fail closed on older targets
> 
> right, without this an outdated remote node would start the VM with tcp
> migration, but the mtunnel endpoint would then die because it only
> allows unix sockets atm..
> 
> >
> > == Backward compatibility approach ==
> >
> > Rather than bumping WS_TUNNEL_VERSION (which would break
> > new-source -> old-target combinations because of the existing
> > "$WS_TUNNEL_VERSION > $tunnel->{version}" check), I'd add a
> > forward-compatible 'caps' field to the version response. Old sources
> > ignore unknown JSON keys; new sources require 'insecure-remote' to be
> > present in caps before allowing migration_type=insecure, and otherwise
> > fall through to the existing WS-tunneled path with no behavioral
> > change.
> 
> what do you think @Fiona?
> 
> > This means all four mix matrices are clean:
> >   - patched <-> patched, secure: identical to today
> >   - unpatched src -> patched tgt: caps ignored, WS path as today
> >   - patched src -> unpatched tgt, secure: caps absent, not checked,
> >     WS path as today
> >   - patched src -> unpatched tgt, insecure: source dies early with a
> >     clear "upgrade target or omit migration_type=insecure" error,
> >     no side effects on target
> >
> > == Security considerations ==
> >
> >   - root-only at the API/CLI layer, same as the local-cluster knob
> >   - documented as requiring trusted/private network between clusters
> 
> this part here is a big one - it really needs documentation that screams
> "double check to ensure this doesn't accidentally broadcast clear text
> migration data over the internet"
> 
> >   - no change to control plane or auth (still API token + TLS fp)
> >   - data plane confidentiality drops to network-layer controls only,
> >     which is identical to the trade-off operators already make for
> >     intra-cluster insecure migration
> >   - no new ports beyond the existing 60000-60050 range that
> >     insecure migration already uses
> >   - source-side caps check ensures no silent downgrade when target
> >     doesn't support it
> >
> > == Open questions ==
> >
> >   1. Is this direction acceptable in principle, or would you prefer
> >      a different direction?
> 
> it mostly looks good to me with a quick glance, it might be sensible to
> wait for additional input by Fiona or Thomas before diving in.
> 
> >   2. Should the 'caps' mechanism be added in a standalone preliminary
> >      patch (useful as future-proofing for any opt-in mtunnel feature),
> >      or rolled into the same series?
> >
> >   3. Should NBD direct-TCP be gated by a separate flag, or is it fine
> >      to have migration_type=insecure imply both RAM and NBD direct?
> >      The intra-cluster knob ties them together today.
> 
> I think it's fine to tie them together, the same considerations apply to
> them.
> 
> >   4. Any preference on the parameter name? I matched migrate_vm
> >      ('migration_type', 'migration_network') for consistency, but
> >      'data-direct-tcp' or similar would also work and arguably be
> >      less misleading since the control plane is still encrypted.
> 
> that's also true for local migration - the data plane is SSH in that
> case..
> 
> 
> 
>