From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9054F1FF0A8 for ; Sat, 17 Oct 2026 16:13:06 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BD2E8214AB; Sat, 17 Oct 2026 16:13:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionescu.at; s=protonmail2; t=1792245990; x=1792505190; bh=Dfo0Qhddz4uxMdwfICOd3acMJlbuihX8Rp0hsP2XmrQ=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=fs11bKk8WYvdyg+yJJJTjmD0kM07kBKcQ6J4IMralN3NmSaX+MNJJYRoW44FnXO6l +vyL0+esxIUXaBKEjW+TQAEC15jpa24u5K9DqtMypbRbbwocUARGol/wp3gJIasssX myW9glJ1cDJwUWlUv098Qfu+MnLzSnEumDGKjrEd2OoB5Wg6ATntp4Jvy2iaVPKTfC i+T9wJAHECDQayIB4ETarRaB0BaKOjgtHYa5MHPfyHoxZgmeZBGKVN2Xw2mT5iqmOx 8iPNTvContiFvFAyaQR+SM0vwBun/zSW0173b0sFYJj7M1ZIE4x6SkdlwVu3DPWNU1 uLBgAN+D4blXg== Date: Sat, 17 Oct 2026 14:06:24 +0000 To: Bogdan Ionescu From: Bogdan Ionescu Subject: Re: [PATCH v3 qemu-server] remote migration: allow insecure TCP data plane Message-ID: In-Reply-To: <20260822124831.15638-1-bogdan@ionescu.at> References: <20260822124831.15638-1-bogdan@ionescu.at> Feedback-ID: 36014335:user:proton X-Pm-Message-ID: 9b1d93287aacdc90ecf134249124c1506c25b76a MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record TO_EQ_FM_DIRECT_MX 0.001 To == From and direct-to-MX Message-ID-Hash: GFRQGWMTKY4DS7XSFX7FHJYD5NVEZBPU X-Message-ID-Hash: GFRQGWMTKY4DS7XSFX7FHJYD5NVEZBPU X-MailFrom: bogdan@ionescu.at X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Fiona, Fabian, Just following up on the v3 patch I sent in August. It implements the two-w= ay capability negotiation suggested by Fiona. Have you had a chance to review it? I'd appreciate any feedback on the appr= oach and any changes needed to move it forward. Thanks, Bogdan On Saturday, August 22nd, 2026 at 3:48 PM, Bogdan Ionescu wrote: > Expose migration_type=3Dinsecure and migration_network for remote > migration. >=20 > The websocket tunnel remains used for control commands, while QEMU > migration state and NBD storage migration can use a direct TCP data > plane. This avoids websocket masking/TLS overhead on trusted private > migration networks. >=20 > Negotiate the opt-in data plane using a dedicated two-way > 'capabilities' mtunnel command. The source sends the capabilities it > wants to use, the target returns the subset it supports and permits, > and both endpoints retain the negotiated set for subsequent handling. > Keep the version command limited to tunnel protocol compatibility. >=20 > Only negotiate capabilities when an opt-in feature is requested, so > regular remote migrations remain compatible with older targets. Fail > closed and clean up the target tunnel if capability negotiation is not > available or the requested capability is not accepted. >=20 > Require Sys.Modify on '/' on the source when selecting the insecure > data plane or an explicit migration network. On the target, only accept > the insecure-remote capability when the remote credentials also have > Sys.Modify on '/'. >=20 > This mode is only intended for fully trusted private networks, as guest > RAM and disk migration data may be transferred in clear text. >=20 > Changes since v2: > - replace one-way capabilities in the version reply with a dedicated > two-way capabilities handler > - keep the negotiated capability set on both source and target > - make target acceptance of insecure-remote conditional on Sys.Modify > - clean up the remote tunnel when capability negotiation fails > - adapt the current mtunnel start schema for tcp/insecure migration > - preserve migration_network for the insecure target start path > - drop the pve-guest-common change; the existing JSON tunnel command > path already carries the capabilities request and reply >=20 > Signed-off-by: Bogdan Ionescu > --- > src/PVE/API2/Qemu.pm | 90 ++++++++++++++++++++++++++---- > src/PVE/CLI/qm.pm | 18 ++++++ > src/PVE/QemuMigrate.pm | 121 ++++++++++++++++++++++++++++++++++------- > 3 files changed, 197 insertions(+), 32 deletions(-) >=20 > diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm > index 71247ee..9eab183 100644 > --- a/src/PVE/API2/Qemu.pm > +++ b/src/PVE/API2/Qemu.pm > @@ -5720,6 +5720,24 @@ __PACKAGE__->register_method({ > minimum =3D> '0', > default =3D> 'migrate limit from datacenter or storage c= onfig', > }, > + migration_type =3D> { > + type =3D> 'string', > + enum =3D> ['secure', 'insecure'], > + description =3D> > + "Migration traffic is encrypted using a websocket tu= nnel by default. " > + . "On secure, completely private networks this can b= e disabled to " > + . "increase performance. WARNING: with 'insecure', V= M RAM and disk " > + . "migration data is transferred in clear text over = the selected " > + . "migration network. The target credentials need Sy= s.Modify on '/'.", > + optional =3D> 1, > + }, > + migration_network =3D> { > + type =3D> 'string', > + format =3D> 'CIDR', > + description =3D> > + "CIDR of the trusted private network used for insecu= re remote migration.", > + optional =3D> 1, > + }, > }, > }, > returns =3D> { > @@ -5735,9 +5753,16 @@ __PACKAGE__->register_method({ > my $source_vmid =3D extract_param($param, 'vmid'); > my $target_endpoint =3D extract_param($param, 'target-endpoint')= ; > my $target_vmid =3D extract_param($param, 'target-vmid') // $sou= rce_vmid; > + my $migration_type =3D extract_param($param, 'migration_type') /= / 'secure'; > + my $migration_network =3D extract_param($param, 'migration_netwo= rk'); >=20 > my $delete =3D extract_param($param, 'delete') // 0; >=20 > + # insecure remote migration can transfer VM RAM and disk data in= clear text > + if ($migration_type eq 'insecure' || defined($migration_network)= ) { > + $rpcenv->check_full($authuser, "/", ['Sys.Modify']); > + } > + > PVE::Cluster::check_cfs_quorum(); >=20 > # test if VM exists > @@ -5837,7 +5862,8 @@ __PACKAGE__->register_method({ > client =3D> $api_client, > vmid =3D> $target_vmid, > }; > - $param->{migration_type} =3D 'websocket'; > + $param->{migration_type} =3D $migration_type eq 'insecure' ? 'in= secure' : 'websocket'; > + $param->{migration_network} =3D $migration_network if defined($m= igration_network); > $param->{'with-local-disks'} =3D 1; > $param->{delete} =3D $delete if $delete; >=20 > @@ -6739,6 +6765,15 @@ __PACKAGE__->register_method({ > }; >=20 > my $cmd_desc =3D { > + capabilities =3D> { > + capabilities =3D> { > + type =3D> 'array', > + items =3D> { > + type =3D> 'string', > + pattern =3D> '[a-z][a-z0-9-]*', > + }, > + }, > + }, > config =3D> { > conf =3D> { > type =3D> 'string', > @@ -6768,7 +6803,7 @@ __PACKAGE__->register_method({ > properties =3D> { > statefile =3D> { > type =3D> 'string', > - enum =3D> ['unix'], > + enum =3D> ['unix', 'tcp'], > }, > forcemachine =3D> get_standard_option('pve-q= emu-machine'), > forcecpu =3D> { > @@ -6793,7 +6828,7 @@ __PACKAGE__->register_method({ > }, > type =3D> { > type =3D> 'string', > - enum =3D> ['websocket'], > + enum =3D> ['websocket', 'insecure'], > }, > remote_node =3D> get_standard_option('pve-no= de'), > network =3D> { > @@ -6851,6 +6886,20 @@ __PACKAGE__->register_method({ > age =3D> 0, > }; > }, > + 'capabilities' =3D> sub { > + my ($params) =3D @_; > + > + my $supported =3D {}; > + $supported->{'insecure-remote'} =3D 1 > + if $rpcenv->check($authuser, '/', ['Sys.Modify']= , 1); > + > + my $requested =3D { map { $_ =3D> 1 } $params->{capa= bilities}->@* }; > + my @accepted =3D grep { $requested->{$_} } sort keys= $supported->%*; > + > + $state->{capabilities} =3D { map { $_ =3D> 1 } @acce= pted }; > + > + return { capabilities =3D> \@accepted }; > + }, > 'config' =3D> sub { > my ($params) =3D @_; >=20 > @@ -6998,6 +7047,17 @@ __PACKAGE__->register_method({ > 'start' =3D> sub { > my ($params) =3D @_; >=20 > + my $migration_type =3D $params->{migrate_opts}->{typ= e}; > + my $statefile =3D $params->{start_params}->{statefil= e}; > + if ($migration_type eq 'insecure') { > + die "insecure remote migration capability was no= t negotiated\n" > + if !($state->{capabilities} // {})->{'insecu= re-remote'}; > + die "insecure remote migration requires a TCP st= atefile\n" > + if $statefile ne 'tcp'; > + } elsif ($statefile ne 'unix') { > + die "websocket remote migration requires a UNIX = statefile\n"; > + } > + > if (my $nbd =3D $params->{migrate_opts}->{nbd}) { > for my $ds (keys $nbd->%*) { > my $drive =3D $nbd->{$ds}; > @@ -7029,7 +7089,7 @@ __PACKAGE__->register_method({ >=20 > # not used, but in schema for compat reasons, could = stop setting it.. > delete $params->{migrate_opts}->{storagemap}; > - delete $params->{migrate_opts}->{network}; > + delete $params->{migrate_opts}->{network} if $migrat= ion_type eq 'websocket'; >=20 > my $info =3D PVE::QemuServer::vm_start_nolock( > $state->{storecfg}, > @@ -7039,19 +7099,25 @@ __PACKAGE__->register_method({ > $params->{migrate_opts}, > ); >=20 > - if ($info->{migrate}->{proto} ne 'unix') { > + my $proto =3D $info->{migrate}->{proto}; > + if ( > + ($migration_type eq 'websocket' && $proto ne 'un= ix') > + || ($migration_type eq 'insecure' && $proto ne '= tcp') > + ) { > PVE::QemuServer::vm_stop(undef, $state->{vmid}, = 1, 1); > - die "migration over non-UNIX sockets not possibl= e\n"; > + die "unexpected migration protocol '$proto' for = '$migration_type' migration\n"; > } >=20 > - my $socket =3D $info->{migrate}->{addr}; > - chown $state->{socket_uid}, -1, $socket; > - $state->{sockets}->{$socket} =3D 1; > - > - my $unix_sockets =3D $info->{migrate}->{unix_sockets= }; > - foreach my $socket (@$unix_sockets) { > + if ($proto eq 'unix') { > + my $socket =3D $info->{migrate}->{addr}; > chown $state->{socket_uid}, -1, $socket; > $state->{sockets}->{$socket} =3D 1; > + > + my $unix_sockets =3D $info->{migrate}->{unix_soc= kets} // []; > + foreach my $socket (@$unix_sockets) { > + chown $state->{socket_uid}, -1, $socket; > + $state->{sockets}->{$socket} =3D 1; > + } > } > return $info; > }, > diff --git a/src/PVE/CLI/qm.pm b/src/PVE/CLI/qm.pm > index b903c1f..cefdf2d 100644 > --- a/src/PVE/CLI/qm.pm > +++ b/src/PVE/CLI/qm.pm > @@ -224,6 +224,24 @@ __PACKAGE__->register_method({ > minimum =3D> '0', > default =3D> 'migrate limit from datacenter or storage c= onfig', > }, > + migration_type =3D> { > + type =3D> 'string', > + enum =3D> ['secure', 'insecure'], > + description =3D> > + "Migration traffic is encrypted using a websocket tu= nnel by default. " > + . "On secure, completely private networks this can b= e disabled to " > + . "increase performance. WARNING: with 'insecure', V= M RAM and disk " > + . "migration data is transferred in clear text over = the selected " > + . "migration network. The target credentials need Sy= s.Modify on '/'.", > + optional =3D> 1, > + }, > + migration_network =3D> { > + type =3D> 'string', > + format =3D> 'CIDR', > + description =3D> > + "CIDR of the trusted private network used for insecu= re remote migration.", > + optional =3D> 1, > + }, > }, > }, > returns =3D> { > diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm > index 8da6f15..c459102 100644 > --- a/src/PVE/QemuMigrate.pm > +++ b/src/PVE/QemuMigrate.pm > @@ -46,6 +46,31 @@ use base qw(PVE::AbstractMigrate); > # compared against remote end's minimum version > our $WS_TUNNEL_VERSION =3D 2; >=20 > +sub negotiate_remote_tunnel_capabilities { > + my ($tunnel, $wanted) =3D @_; > + > + return {} if !@$wanted; > + > + my $res =3D eval { > + PVE::Tunnel::write_tunnel( > + $tunnel, 10, 'capabilities', { capabilities =3D> $wanted }, > + ); > + }; > + die "failed to negotiate remote migration capabilities - $@" if $@; > + die "invalid capabilities response from remote tunnel endpoint\n" > + if ref($res) ne 'HASH' || ref($res->{capabilities}) ne 'ARRAY'; > + > + my $wanted_caps =3D { map { $_ =3D> 1 } @$wanted }; > + my $accepted =3D {}; > + for my $cap ($res->{capabilities}->@*) { > + die "remote tunnel endpoint returned unrequested capability '$ca= p'\n" > + if !$wanted_caps->{$cap}; > + $accepted->{$cap} =3D 1; > + } > + > + return $accepted; > +} > + > sub fork_tunnel { > my ($self, $ssh_forward_info) =3D @_; >=20 > @@ -352,6 +377,33 @@ sub prepare { > die "Remote tunnel endpoint too old, upgrade required\n" > if $WS_TUNNEL_VERSION > $tunnel->{version}; >=20 > + my @wanted_capabilities; > + push @wanted_capabilities, 'insecure-remote' > + if $self->{opts}->{migration_type} eq 'insecure'; > + > + my $capabilities =3D eval { > + negotiate_remote_tunnel_capabilities($tunnel, \@wanted_capab= ilities); > + }; > + my $capability_err =3D $@; > + > + if ( > + $capability_err > + || ($self->{opts}->{migration_type} eq 'insecure' > + && !$capabilities->{'insecure-remote'}) > + ) { > + eval { PVE::Tunnel::finish_tunnel($tunnel, 1); }; > + my $cleanup_err =3D $@; > + > + my $err =3D "Remote tunnel endpoint did not accept insecure = remote migration; " > + . "upgrade target, grant Sys.Modify on '/' to the target= credentials, or omit " > + . "migration_type=3Dinsecure"; > + $err .=3D " - $capability_err" if $capability_err; > + $err .=3D " (tunnel cleanup failed: $cleanup_err)" if $clean= up_err; > + die "$err\n"; > + } > + > + $tunnel->{capabilities} =3D $capabilities; > + > print "websocket tunnel started\n"; > $self->{tunnel} =3D $tunnel; > } else { > @@ -1150,8 +1202,9 @@ sub phase2_start_local_cluster { > sub phase2_start_remote_cluster { > my ($self, $vmid, $params) =3D @_; >=20 > - die "insecure migration to remote cluster not implemented\n" > - if $params->{migrate_opts}->{type} ne 'websocket'; > + die "unsupported remote migration type '$params->{migrate_opts}->{ty= pe}'\n" > + if $params->{migrate_opts}->{type} ne 'websocket' > + && $params->{migrate_opts}->{type} ne 'insecure'; >=20 > my $remote_vmid =3D $self->{opts}->{remote}->{vmid}; >=20 > @@ -1165,8 +1218,14 @@ sub phase2_start_remote_cluster { > $self->{stopnbd} =3D 1; > $self->{target_drive}->{$drive}->{drivestr} =3D $res->{drives}->= {$drive}->{drivestr}; > my $nbd_uri =3D $res->{drives}->{$drive}->{nbd_uri}; > - die "unexpected NBD uri for '$drive': $nbd_uri\n" > - if $nbd_uri !~ s!/run/qemu-server/$remote_vmid\_!/run/qemu-s= erver/$vmid\_!; > + if ($params->{migrate_opts}->{type} eq 'websocket') { > + die "unexpected NBD uri for '$drive': $nbd_uri\n" > + if $nbd_uri !~ s!/run/qemu-server/$remote_vmid\_!/run/qe= mu-server/$vmid\_!; > + } elsif ($params->{migrate_opts}->{type} eq 'insecure') { > + die "unexpected NBD uri for '$drive': $nbd_uri\n" > + if $nbd_uri > + !~ m!^nbd:(?:localhost|[\d\.]+|\[[\d\.:a-fA-F]+\]):\d+:e= xportname=3Ddrive-[A-Za-z0-9_.-]+$!; > + } >=20 > $self->{target_drive}->{$drive}->{nbd_uri} =3D $nbd_uri; > } > @@ -1260,28 +1319,50 @@ sub phase2 { > my $remote_vmid =3D $remote->{vmid}; > $params->{migrate_opts}->{remote_node} =3D $self->{node}; > ($tunnel_info, $spice_port) =3D $self->phase2_start_remote_clust= er($vmid, $params); > - die "only UNIX sockets are supported for remote migration\n" > - if $tunnel_info->{proto} ne 'unix'; > - > - # untaint > - my ($remote_socket) =3D $tunnel_info->{addr} =3D~ m|^(/run/qemu-= server/\d+\.migrate)$| > - or die "unexpected socket address '$tunnel_info->{addr}'\n"; > - my $local_socket =3D $remote_socket; > - $local_socket =3D~ s/$remote_vmid/$vmid/g; > - $tunnel_info->{addr} =3D $local_socket; > - > - $self->log('info', "Setting up tunnel for '$local_socket'"); > - PVE::Tunnel::forward_unix_socket($self->{tunnel}, $local_socket,= $remote_socket); > + if ($params->{migrate_opts}->{type} eq 'websocket') { > + die "only UNIX sockets are supported for remote migration\n" > + if $tunnel_info->{proto} ne 'unix'; >=20 > - foreach my $remote_socket (@{ $tunnel_info->{unix_sockets} }) { > # untaint > - ($remote_socket) =3D $remote_socket =3D~ m|^(/run/qemu-serve= r/(?:(?!\.\./).)+\.migrate)$| > - or die "unexpected socket address '$remote_socket'\n"; > + my ($remote_socket) =3D $tunnel_info->{addr} =3D~ m|^(/run/q= emu-server/\d+\.migrate)$| > + or die "unexpected socket address '$tunnel_info->{addr}'= \n"; > my $local_socket =3D $remote_socket; > $local_socket =3D~ s/$remote_vmid/$vmid/g; > - next if $self->{tunnel}->{forwarded}->{$local_socket}; > + $tunnel_info->{addr} =3D $local_socket; > + > $self->log('info', "Setting up tunnel for '$local_socket'"); > PVE::Tunnel::forward_unix_socket($self->{tunnel}, $local_soc= ket, $remote_socket); > + > + foreach my $remote_socket (@{ $tunnel_info->{unix_sockets} /= / [] }) { > + # untaint > + ($remote_socket) =3D > + $remote_socket =3D~ m|^(/run/qemu-server/(?:(?!\.\./= ).)+\.migrate)$| > + or die "unexpected socket address '$remote_socket'\n= "; > + my $local_socket =3D $remote_socket; > + $local_socket =3D~ s/$remote_vmid/$vmid/g; > + next if $self->{tunnel}->{forwarded}->{$local_socket}; > + $self->log('info', "Setting up tunnel for '$local_socket= '"); > + PVE::Tunnel::forward_unix_socket($self->{tunnel}, $local= _socket, $remote_socket); > + } > + } elsif ($params->{migrate_opts}->{type} eq 'insecure') { > + die "only TCP sockets are supported for insecure remote migr= ation\n" > + if $tunnel_info->{proto} ne 'tcp'; > + > + die "unexpected TCP migration address '$tunnel_info->{addr}'= \n" > + if $tunnel_info->{addr} !~ m/^(?:localhost|[\d\.]+|\[[\d= \.:a-fA-F]+\])$/; > + > + die "unexpected TCP migration port '$tunnel_info->{port}'\n" > + if !defined($tunnel_info->{port}) > + || $tunnel_info->{port} !~ /^\d+$/ > + || $tunnel_info->{port} <=3D 0 > + || $tunnel_info->{port} > 65535; > + > + $self->log( > + 'info', > + "using direct TCP migration to $tunnel_info->{addr}:$tun= nel_info->{port}", > + ); > + } else { > + die "unsupported remote migration type '$params->{migrate_op= ts}->{type}'\n"; > } > } else { > ($tunnel_info, $spice_port) =3D $self->phase2_start_local_cluste= r($vmid, $params); > base-commit: e6352be67f70042a7433a3a3c712b36d02f9f7cb > -- > 2.47.3 >=20 >