From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 4BABE1FF13C for ; Thu, 30 Apr 2026 14:40:20 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 256C95AC4; Thu, 30 Apr 2026 14:40:20 +0200 (CEST) Date: Thu, 30 Apr 2026 14:40:09 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [pve-devel] [RFC] qemu-server: add migration_type=insecure to remote-migrate To: Bogdan Ionescu , "pve-devel@lists.proxmox.com" References: In-Reply-To: MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1777552058.4o39hpnqt6.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1777552713886 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.054 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RMB5ETEWXUW5CJNZLHWXVXQQST226PA5 X-Message-ID-Hash: RMB5ETEWXUW5CJNZLHWXVXQQST226PA5 X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On April 25, 2026 3:10 am, Bogdan Ionescu wrote: > Hi all, >=20 > I'd like to gauge interest in adding a migration_type=3Dinsecure option t= o > the qm remote-migrate / remote_migrate_vm endpoint, before investing > time in a review-ready patch series. Hi! This is something that we will need sooner or later as well, in the context of PDM and fabrics. > =3D=3D Motivation =3D=3D >=20 > The current remote-migrate implementation tunnels both control plane > and data plane through the websocket connection to the target's API > endpoint on 8006/tcp. This is the right default for trust reasons > (API token + TLS fingerprint, no SSH trust between clusters needed), > but the data plane throughput is severely bottlenecked by: >=20 > - userspace bouncing through PVE::Tunnel + pveproxy + qmtunnel > (3 Perl processes in the data path, each context-switching per > chunk) > - per-byte WebSocket masking in pure Perl (RFC 6455 =C2=A75.3) > - TLS framing on top > - lack of zero-copy / TSO offload for the streamed bytes > - multiple TCP segments end-to-end with independent flow control >=20 > In our deployment between two DCs connected by WireGuard over a > 10 Gbps link, we observe sustained ~1 MB/s for remote-migrate while > intra-cluster `qm migrate --migration_type insecure` between the same > hosts saturates the link at ~300+ MB/s. The bottleneck is clearly > the WS tunnel data path on a single Perl-bound core, not the network. >=20 > For VMs with 32+ GB of RAM, this difference is the difference between > a migration finishing in 2 minutes vs. failing to converge entirely > because the dirty rate exceeds the throughput. >=20 > =3D=3D Proposal =3D=3D >=20 > Mirror the local-cluster migration model: keep secure (WS-tunneled) as > the default, allow opt-in 'insecure' for trusted networks where the > operator has out-of-band guarantees (private cross-connect, VPN, > overlay encryption at L2/L3). >=20 > qm remote-migrate 'apitoken=3D...,host=3D...,fp=3D= ...' \ > --target-storage ... --target-bridge ... --online \ > --migration_type insecure \ > --migration_network 10.50.0.0/24 >=20 > Semantics: > - control plane (config, NBD allocation requests, tunnel commands, > spice ticket, etc.) still goes through the WS tunnel as today this makes sense > - data plane (QEMU memory stream + NBD storage drive-mirror) goes > direct TCP between source and target on the standard > 60000-60050 range, with the target's listener IP resolved from > --migration_network (same logic as local-cluster insecure) this as well, though as an alternative one might consider providing an interface name as well? > - root-only on the source side, consistent with migrate_vm here we have a slight difference between intra-cluster and inter-cluster migrations: - within a cluster, we have established trust and a shared authentication scope - node A asking node B about its migration address is okay (post-authentication), since a regular user cannot override it - between clusters, we have less guarantees - while the target has to trust the source somewhat (which is why we require a separate privilege for allowing incoming remote migrations in the first place), I am not sure whether we would not want to require some additional privileges for allowing insecure migrations as well? e.g. Sys.Modify somewhere, or something similar? we might also consider whether it makes sense to pre-configure remote migration networks and allow selecting them by ID, though that could be added later as follow-up as well. > - target advertises an 'insecure-remote' capability in the mtunnel > version response so source can fail closed on older targets right, without this an outdated remote node would start the VM with tcp migration, but the mtunnel endpoint would then die because it only allows unix sockets atm.. >=20 > =3D=3D Backward compatibility approach =3D=3D >=20 > Rather than bumping WS_TUNNEL_VERSION (which would break > new-source -> old-target combinations because of the existing > "$WS_TUNNEL_VERSION > $tunnel->{version}" check), I'd add a > forward-compatible 'caps' field to the version response. Old sources > ignore unknown JSON keys; new sources require 'insecure-remote' to be > present in caps before allowing migration_type=3Dinsecure, and otherwise > fall through to the existing WS-tunneled path with no behavioral > change. what do you think @Fiona? > This means all four mix matrices are clean: > - patched <-> patched, secure: identical to today > - unpatched src -> patched tgt: caps ignored, WS path as today > - patched src -> unpatched tgt, secure: caps absent, not checked, > WS path as today > - patched src -> unpatched tgt, insecure: source dies early with a > clear "upgrade target or omit migration_type=3Dinsecure" error, > no side effects on target >=20 > =3D=3D Security considerations =3D=3D >=20 > - root-only at the API/CLI layer, same as the local-cluster knob > - documented as requiring trusted/private network between clusters this part here is a big one - it really needs documentation that screams "double check to ensure this doesn't accidentally broadcast clear text migration data over the internet" > - no change to control plane or auth (still API token + TLS fp) > - data plane confidentiality drops to network-layer controls only, > which is identical to the trade-off operators already make for > intra-cluster insecure migration > - no new ports beyond the existing 60000-60050 range that > insecure migration already uses > - source-side caps check ensures no silent downgrade when target > doesn't support it >=20 > =3D=3D Open questions =3D=3D >=20 > 1. Is this direction acceptable in principle, or would you prefer > a different direction? it mostly looks good to me with a quick glance, it might be sensible to wait for additional input by Fiona or Thomas before diving in. > 2. Should the 'caps' mechanism be added in a standalone preliminary > patch (useful as future-proofing for any opt-in mtunnel feature), > or rolled into the same series? >=20 > 3. Should NBD direct-TCP be gated by a separate flag, or is it fine > to have migration_type=3Dinsecure imply both RAM and NBD direct? > The intra-cluster knob ties them together today. I think it's fine to tie them together, the same considerations apply to them. > 4. Any preference on the parameter name? I matched migrate_vm > ('migration_type', 'migration_network') for consistency, but > 'data-direct-tcp' or similar would also work and arguably be > less misleading since the control plane is still encrypted. that's also true for local migration - the data plane is SSH in that case..