From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id F196D920F3 for ; Mon, 3 Oct 2022 09:12:37 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D62942E3EE for ; Mon, 3 Oct 2022 09:12:07 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 3 Oct 2022 09:12:04 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C36FA444A8 for ; Mon, 3 Oct 2022 09:11:57 +0200 (CEST) Date: Mon, 03 Oct 2022 09:11:50 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox VE development discussion References: <20220928125059.1139296-1-f.gruenbichler@proxmox.com> <20220928125059.1139296-9-f.gruenbichler@proxmox.com> <7e10feea-3c99-232b-b556-8361b95886a8@proxmox.com> In-Reply-To: <7e10feea-3c99-232b-b556-8361b95886a8@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1664780846.jd1ah2uiwp.astroid@nora.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.007 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH v6 qemu-server 2/6] mtunnel: add API endpoints X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2022 07:12:38 -0000 On September 30, 2022 1:52 pm, Stefan Hanreich wrote: >=20 >=20 > On 9/28/22 14:50, Fabian Gr=C3=BCnbichler wrote: >> the following two endpoints are used for migration on the remote side >>=20 >> POST /nodes/NODE/qemu/VMID/mtunnel >>=20 >> which creates and locks an empty VM config, and spawns the main qmtunnel >> worker which binds to a VM-specific UNIX socket. >>=20 >> this worker handles JSON-encoded migration commands coming in via this >> UNIX socket: >> - config (set target VM config) >> -- checks permissions for updating config >> -- strips pending changes and snapshots >> -- sets (optional) firewall config >> - disk (allocate disk for NBD migration) >> -- checks permission for target storage >> -- returns drive string for allocated volume >> - disk-import, query-disk-import, bwlimit >> -- handled by PVE::StorageTunnel >> - start (returning migration info) >> - fstrim (via agent) >> - ticket (creates a ticket for a WS connection to a specific socket) >> - resume >> - stop >> - nbdstop >> - unlock >> - quit (+ cleanup) >>=20 >> this worker serves as a replacement for both 'qm mtunnel' and various >> manual calls via SSH. the API call will return a ticket valid for >> connecting to the worker's UNIX socket via a websocket connection. >>=20 >> GET+WebSocket upgrade /nodes/NODE/qemu/VMID/mtunnelwebsocket >>=20 >> gets called for connecting to a UNIX socket via websocket forwarding, >> i.e. once for the main command mtunnel, and once each for the memory >> migration and each NBD drive-mirror/storage migration. >>=20 >> access is guarded by a short-lived ticket binding the authenticated user >> to the socket path. such tickets can be requested over the main mtunnel, >> which keeps track of socket paths currently used by that >> mtunnel/migration instance. >>=20 >> each command handler should check privileges for the requested action if >> necessary. >>=20 >> both mtunnel and mtunnelwebsocket endpoints are not proxied, the >> client/caller is responsible for ensuring the passed 'node' parameter >> and the endpoint handling the call are matching. >>=20 >> Signed-off-by: Fabian Gr=C3=BCnbichler >> --- >>=20 >> Notes: >> v6: >> - check for Sys.Incoming in mtunnel >> - add definedness checks in 'config' command >> - switch to vm_running_locally in 'resume' command >> - moved $socket_addr closer to usage >> v5: >> - us vm_running_locally >> - move '$socket_addr' declaration closer to usage >> v4: >> - add timeout to accept() >> - move 'bwlimit' to PVE::StorageTunnel and extend it >> - mark mtunnel(websocket) as non-proxied, and check $node according= ly >> v3: >> - handle meta and vmgenid better >> - handle failure of 'config' updating >> - move 'disk-import' and 'query-disk-import' handlers to pve-guest-= common >> - improve tunnel exit by letting client close the connection >> - use strict VM config parser >> v2: incorporated Fabian Ebner's feedback, mainly: >> - use modified nbd alloc helper instead of duplicating >> - fix disk cleanup, also cleanup imported disks >> - fix firewall-conf vs firewall-config mismatch >> =20 >> requires >> - pve-access-control with tunnel ticket support (already marked in = d/control) >> - pve-access-control with Sys.Incoming privilege (not yet applied/b= umped!) >> - pve-http-server with websocket fixes (could be done via breaks? o= r bumped in >> pve-manager..) >>=20 >> PVE/API2/Qemu.pm | 527 ++++++++++++++++++++++++++++++++++++++++++++++- >> debian/control | 2 +- >> 2 files changed, 527 insertions(+), 2 deletions(-) >>=20 >> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm >> index 3ec31c26..9270ca74 100644 >> --- a/PVE/API2/Qemu.pm >> +++ b/PVE/API2/Qemu.pm >> @@ -4,10 +4,13 @@ use strict; >> use warnings; >> use Cwd 'abs_path'; >> use Net::SSLeay; >> -use POSIX; >> use IO::Socket::IP; >> +use IO::Socket::UNIX; >> +use IPC::Open3; >> +use JSON; >> use URI::Escape; >> use Crypt::OpenSSL::Random; >> +use Socket qw(SOCK_STREAM); >> =20 >> use PVE::Cluster qw (cfs_read_file cfs_write_file);; >> use PVE::RRD; >> @@ -38,6 +41,7 @@ use PVE::VZDump::Plugin; >> use PVE::DataCenterConfig; >> use PVE::SSHInfo; >> use PVE::Replication; >> +use PVE::StorageTunnel; >> =20 >> BEGIN { >> if (!$ENV{PVE_GENERATING_DOCS}) { >> @@ -1087,6 +1091,7 @@ __PACKAGE__->register_method({ >> { subdir =3D> 'spiceproxy' }, >> { subdir =3D> 'sendkey' }, >> { subdir =3D> 'firewall' }, >> + { subdir =3D> 'mtunnel' }, >> ]; >> =20 >> return $res; >> @@ -4965,4 +4970,524 @@ __PACKAGE__->register_method({ >> return PVE::QemuServer::Cloudinit::dump_cloudinit_config($conf, $para= m->{vmid}, $param->{type}); >> }}); >> =20 >> +__PACKAGE__->register_method({ >> + name =3D> 'mtunnel', >> + path =3D> '{vmid}/mtunnel', >> + method =3D> 'POST', >> + protected =3D> 1, >> + description =3D> 'Migration tunnel endpoint - only for internal use= by VM migration.', >> + permissions =3D> { >> + check =3D> >> + [ 'and', >> + ['perm', '/vms/{vmid}', [ 'VM.Allocate' ]], >> + ['perm', '/', [ 'Sys.Incoming' ]], >> + ], >> + description =3D> "You need 'VM.Allocate' permissions on '/vms/{vmid}' = and Sys.Incoming" . >> + " on '/'. Further permission checks happen during the a= ctual migration.", >> + }, >> + parameters =3D> { >> + additionalProperties =3D> 0, >> + properties =3D> { >> + node =3D> get_standard_option('pve-node'), >> + vmid =3D> get_standard_option('pve-vmid'), >> + storages =3D> { >> + type =3D> 'string', >> + format =3D> 'pve-storage-id-list', >> + optional =3D> 1, >> + description =3D> 'List of storages to check permission and availabili= ty. Will be checked again for all actually used storages during migration.'= , >> + }, >> + }, >> + }, >> + returns =3D> { >> + additionalProperties =3D> 0, >> + properties =3D> { >> + upid =3D> { type =3D> 'string' }, >> + ticket =3D> { type =3D> 'string' }, >> + socket =3D> { type =3D> 'string' }, >> + }, >> + }, >> + code =3D> sub { [...] >> + my $cmd =3D delete $parsed->{cmd}; >> + if (!defined($cmd)) { >> + $reply_err->("'cmd' missing"); >> + } elsif ($state->{exit}) { >> + $reply_err->("tunnel is in exit-mode, processing '$cmd' cmd not p= ossible"); >> + next; >> + } elsif (my $handler =3D $cmd_handlers->{$cmd}) { >> + print "received command '$cmd'\n"; >> + eval { >> + if ($cmd_desc->{$cmd}) { >> + PVE::JSONSchema::validate($cmd_desc->{$cmd}, $parsed); >=20 > might the params be flipped here? >=20 yes! thanks for catching (and wow - our schema handling is flexible that=20 it never choked on that!). I'll do some tests and see whether flipping breaks anything (the same is=20 also true for pve-container, since this part is duplicated there). >> + } else { >> + $parsed =3D {}; >> + } >> + my $res =3D $run_locked->($handler, $parsed); >> + $reply_ok->($res); >> + }; >> + $reply_err->("failed to handle '$cmd' command - $@") >> + if $@; >> + } else { >> + $reply_err->("unknown command '$cmd' given"); >> + } >> + } >> + >> + if ($state->{exit}) { >> + print "mtunnel exited\n"; >> + } else { >> + die "mtunnel exited unexpectedly\n"; >> + } >> + }; >> + >> + my $socket_addr =3D "/run/qemu-server/$vmid.mtunnel"; >> + my $ticket =3D PVE::AccessControl::assemble_tunnel_ticket($authuser, "= /socket/$socket_addr"); >> + my $upid =3D $rpcenv->fork_worker('qmtunnel', $vmid, $authuser, $realc= md); >> + >> + return { >> + ticket =3D> $ticket, >> + upid =3D> $upid, >> + socket =3D> $socket_addr, >> + }; >> + }}); >> + >> +__PACKAGE__->register_method({ >> + name =3D> 'mtunnelwebsocket', >> + path =3D> '{vmid}/mtunnelwebsocket', >> + method =3D> 'GET', >> + permissions =3D> { >> + description =3D> "You need to pass a ticket valid for the selected soc= ket. Tickets can be created via the mtunnel API call, which will check perm= issions accordingly.", >> + user =3D> 'all', # check inside >> + }, >> + description =3D> 'Migration tunnel endpoint for websocket upgrade -= only for internal use by VM migration.', >> + parameters =3D> { >> + additionalProperties =3D> 0, >> + properties =3D> { >> + node =3D> get_standard_option('pve-node'), >> + vmid =3D> get_standard_option('pve-vmid'), >> + socket =3D> { >> + type =3D> "string", >> + description =3D> "unix socket to forward to", >> + }, >> + ticket =3D> { >> + type =3D> "string", >> + description =3D> "ticket return by initial 'mtunnel' API call, or ret= rieved via 'ticket' tunnel command", >> + }, >> + }, >> + }, >> + returns =3D> { >> + type =3D> "object", >> + properties =3D> { >> + port =3D> { type =3D> 'string', optional =3D> 1 }, >> + socket =3D> { type =3D> 'string', optional =3D> 1 }, >> + }, >> + }, >> + code =3D> sub { >> + my ($param) =3D @_; >> + >> + my $rpcenv =3D PVE::RPCEnvironment::get(); >> + my $authuser =3D $rpcenv->get_user(); >> + >> + my $nodename =3D PVE::INotify::nodename(); >> + my $node =3D extract_param($param, 'node'); >> + >> + raise_param_exc({ node =3D> "node needs to be 'localhost' or local hos= tname '$nodename'" }) >> + if $node ne 'localhost' && $node ne $nodename; >> + >> + my $vmid =3D $param->{vmid}; >> + # check VM exists >> + PVE::QemuConfig->load_config($vmid); >> + >> + my $socket =3D $param->{socket}; >> + PVE::AccessControl::verify_tunnel_ticket($param->{ticket}, $authuser, = "/socket/$socket"); >> + >> + return { socket =3D> $socket }; >> + }}); >> + >> 1; >> diff --git a/debian/control b/debian/control >> index a90ecd6f..ce469cbd 100644 >> --- a/debian/control >> +++ b/debian/control >> @@ -33,7 +33,7 @@ Depends: dbus, >> libjson-perl, >> libjson-xs-perl, >> libnet-ssleay-perl, >> - libpve-access-control (>=3D 5.0-7), >> + libpve-access-control (>=3D 7.0-7), >> libpve-cluster-perl, >> libpve-common-perl (>=3D 7.1-4), >> libpve-guest-common-perl (>=3D 4.1-1), >=20 >=20 > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >=20