From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id B2BC91FF14C for ; Fri, 15 May 2026 00:26:59 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B4572514D; Fri, 15 May 2026 00:26:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionescu.at; s=protonmail; t=1778797552; x=1779056752; bh=6G3d+AIMFToj0PKNS/qzIzPaMl6dE7oy8BQj2vPV8+0=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=kh1LfdHl/121DRH7mx09CRmcnL+xsIa3b5v9SXUcOMOLoX9zcPxRVU9lcVfRBODiG xgqDAQ6GIETIDtciYA03itqcXYi11ZrU9vVRRkflBMB3nftE9V9ShHX3nFzlwlGyUN CZllPfwuNu5mwGg22PrNxlVSG0XPISeUoGqgXcvgnWmxW1A4ZWhm4CCNK9RTzLf4Wj pOArSKRkW/nxtaQCZBP+lZDYiXtqWK0/a8ntGqybe9bI4gfdfWmfxu3fPDhhiqmxv8 4nmax0w17hixAEMCbz8OfcN1oSc3fqAM3uV9u5N5hqv4eF9hGfkSMbLedTcgI9QgO6 rOk/2LDb/37UA== Date: Thu, 14 May 2026 22:25:45 +0000 To: "pve-devel@lists.proxmox.com" From: Bogdan Ionescu Subject: Re: [pve-devel] [RFC] qemu-server: add migration_type=insecure to remote-migrate Message-ID: In-Reply-To: <1777552058.4o39hpnqt6.astroid@yuna.none> References: <1777552058.4o39hpnqt6.astroid@yuna.none> Feedback-ID: 36014335:user:proton X-Pm-Message-ID: df0b48265d214ae35a4b6e28613a7ffc73503ae8 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: R4X3K3NTGDX5BR6HEW27ZRUAI6HLMO77 X-Message-ID-Hash: R4X3K3NTGDX5BR6HEW27ZRUAI6HLMO77 X-MailFrom: bogdan@ionescu.at X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, following the feedback, I reworked the implementation and split it into a s= mall patch series. Changes compared to the RFC version: kept the websocket tunnel for the control plane only migration and NBD storage traffic use direct TCP only when migration_type= =3Dinsecure added capability-based negotiation (insecure-remote) instead of bumping the= tunnel API version added support for migration_type / migration_network to qm remote-migrate added additional permission checks requiring Sys.Modify on / added more explicit warnings that guest RAM and disk migration data may be = transferred unencrypted I tested this with online remote migration between two PVE 9 nodes using lo= cal ZFS-backed disks. The remote migration path successfully used: NBD over direct TCP for storage migration direct TCP for QEMU migration state websocket tunnel only for the control plane Patch series follows. Kind regards,=C2=A0 Bogdan On Thursday, April 30th, 2026 at 3:40 PM, Fabian Gr=C3=BCnbichler wrote: > On April 25, 2026 3:10 am, Bogdan Ionescu wrote: > > Hi all, > > > > I'd like to gauge interest in adding a migration_type=3Dinsecure option= to > > the qm remote-migrate / remote_migrate_vm endpoint, before investing > > time in a review-ready patch series. >=20 > Hi! >=20 > This is something that we will need sooner or later as well, in the > context of PDM and fabrics. >=20 > > =3D=3D Motivation =3D=3D > > > > The current remote-migrate implementation tunnels both control plane > > and data plane through the websocket connection to the target's API > > endpoint on 8006/tcp. This is the right default for trust reasons > > (API token + TLS fingerprint, no SSH trust between clusters needed), > > but the data plane throughput is severely bottlenecked by: > > > > - userspace bouncing through PVE::Tunnel + pveproxy + qmtunnel > > (3 Perl processes in the data path, each context-switching per > > chunk) > > - per-byte WebSocket masking in pure Perl (RFC 6455 =C2=A75.3) > > - TLS framing on top > > - lack of zero-copy / TSO offload for the streamed bytes > > - multiple TCP segments end-to-end with independent flow control > > > > In our deployment between two DCs connected by WireGuard over a > > 10 Gbps link, we observe sustained ~1 MB/s for remote-migrate while > > intra-cluster `qm migrate --migration_type insecure` between the same > > hosts saturates the link at ~300+ MB/s. The bottleneck is clearly > > the WS tunnel data path on a single Perl-bound core, not the network. > > > > For VMs with 32+ GB of RAM, this difference is the difference between > > a migration finishing in 2 minutes vs. failing to converge entirely > > because the dirty rate exceeds the throughput. > > > > =3D=3D Proposal =3D=3D > > > > Mirror the local-cluster migration model: keep secure (WS-tunneled) as > > the default, allow opt-in 'insecure' for trusted networks where the > > operator has out-of-band guarantees (private cross-connect, VPN, > > overlay encryption at L2/L3). > > > > qm remote-migrate 'apitoken=3D...,host=3D...,fp= =3D...' \ > > --target-storage ... --target-bridge ... --online \ > > --migration_type insecure \ > > --migration_network 10.50.0.0/24 > > > > Semantics: > > - control plane (config, NBD allocation requests, tunnel commands, > > spice ticket, etc.) still goes through the WS tunnel as today >=20 > this makes sense >=20 > > - data plane (QEMU memory stream + NBD storage drive-mirror) goes > > direct TCP between source and target on the standard > > 60000-60050 range, with the target's listener IP resolved from > > --migration_network (same logic as local-cluster insecure) >=20 > this as well, though as an alternative one might consider providing an > interface name as well? >=20 > > - root-only on the source side, consistent with migrate_vm >=20 > here we have a slight difference between intra-cluster and inter-cluster > migrations: > - within a cluster, we have established trust and a shared > authentication scope - node A asking node B about its migration > address is okay (post-authentication), since a regular user cannot > override it > - between clusters, we have less guarantees - while the target has to > trust the source somewhat (which is why we require a separate > privilege for allowing incoming remote migrations in the first place), > I am not sure whether we would not want to require some additional > privileges for allowing insecure migrations as well? e.g. Sys.Modify > somewhere, or something similar? >=20 > we might also consider whether it makes sense to pre-configure remote > migration networks and allow selecting them by ID, though that could be > added later as follow-up as well. >=20 > > - target advertises an 'insecure-remote' capability in the mtunnel > > version response so source can fail closed on older targets >=20 > right, without this an outdated remote node would start the VM with tcp > migration, but the mtunnel endpoint would then die because it only > allows unix sockets atm.. >=20 > > > > =3D=3D Backward compatibility approach =3D=3D > > > > Rather than bumping WS_TUNNEL_VERSION (which would break > > new-source -> old-target combinations because of the existing > > "$WS_TUNNEL_VERSION > $tunnel->{version}" check), I'd add a > > forward-compatible 'caps' field to the version response. Old sources > > ignore unknown JSON keys; new sources require 'insecure-remote' to be > > present in caps before allowing migration_type=3Dinsecure, and otherwis= e > > fall through to the existing WS-tunneled path with no behavioral > > change. >=20 > what do you think @Fiona? >=20 > > This means all four mix matrices are clean: > > - patched <-> patched, secure: identical to today > > - unpatched src -> patched tgt: caps ignored, WS path as today > > - patched src -> unpatched tgt, secure: caps absent, not checked, > > WS path as today > > - patched src -> unpatched tgt, insecure: source dies early with a > > clear "upgrade target or omit migration_type=3Dinsecure" error, > > no side effects on target > > > > =3D=3D Security considerations =3D=3D > > > > - root-only at the API/CLI layer, same as the local-cluster knob > > - documented as requiring trusted/private network between clusters >=20 > this part here is a big one - it really needs documentation that screams > "double check to ensure this doesn't accidentally broadcast clear text > migration data over the internet" >=20 > > - no change to control plane or auth (still API token + TLS fp) > > - data plane confidentiality drops to network-layer controls only, > > which is identical to the trade-off operators already make for > > intra-cluster insecure migration > > - no new ports beyond the existing 60000-60050 range that > > insecure migration already uses > > - source-side caps check ensures no silent downgrade when target > > doesn't support it > > > > =3D=3D Open questions =3D=3D > > > > 1. Is this direction acceptable in principle, or would you prefer > > a different direction? >=20 > it mostly looks good to me with a quick glance, it might be sensible to > wait for additional input by Fiona or Thomas before diving in. >=20 > > 2. Should the 'caps' mechanism be added in a standalone preliminary > > patch (useful as future-proofing for any opt-in mtunnel feature), > > or rolled into the same series? > > > > 3. Should NBD direct-TCP be gated by a separate flag, or is it fine > > to have migration_type=3Dinsecure imply both RAM and NBD direct? > > The intra-cluster knob ties them together today. >=20 > I think it's fine to tie them together, the same considerations apply to > them. >=20 > > 4. Any preference on the parameter name? I matched migrate_vm > > ('migration_type', 'migration_network') for consistency, but > > 'data-direct-tcp' or similar would also work and arguably be > > less misleading since the control plane is still encrypted. >=20 > that's also true for local migration - the data plane is SSH in that > case.. >=20 >=20 >=20 >